2024-02-19T21:51:57.769 INFO:root:teuthology version: 0.0.1.dev260+gb1dac55 2024-02-19T21:51:57.769 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --verbose --archive /home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566803 --name yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566803/orig.config.yaml 2024-02-19T21:51:57.818 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-19T21:51:57.903 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566803 branch: pacific-release description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: null first_in_suite: false job_id: '7566803' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: pacific-release ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(PG_DEGRADED\) - Degraded data redundancy - \(PG_ - MDS_INSUFFICIENT_STANDBY - deprecated feature inline_data - compat changed unexpectedly - overall HEALTH_ - OSD_DOWN - OSD_ - but it is still running - is not responding sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: pacific-release sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 9538 sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 sleep_before_teardown: 0 subset: 111/120000 suite: fs suite_branch: pacific-release suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 targets: smithi052.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK3gdrbO5novp6qfB1dH4FvuFEW8gqbmsgtUnuKo6FK5q5Z2YsmKPlBawQApgt+K1xXvoIYa9OXRPIL808vgEs4= smithi088.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJsfHsnFkPbhk/aA3BZaiEoJv0bulu+9B1Y4Z9juUllAkaC2GW4G+K0qS9Zxw44kth7QvmHbxwlmvRgZ3K2bAes= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - pexec: clients: - sudo modprobe -r ceph - sudo modprobe ceph disable_send_metrics=on - install: exclude_packages: - ceph-volume tag: v16.2.4 - print: '**** done install task...' - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: b1dac5519c57c269ea98a22fb7729016a1d0e4d2 timestamp: 2024-02-19_19:27:49 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-02-19T21:51:57.903 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa; will attempt to use it 2024-02-19T21:51:57.904 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks 2024-02-19T21:51:57.904 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-19T21:51:57.905 INFO:teuthology.task.internal:Checking packages... 2024-02-19T21:51:57.929 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '81bd20d634209c7cb82c18be12b4b5a05643ebf1' 2024-02-19T21:51:57.929 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-19T21:51:57.929 INFO:teuthology.packaging:ref: None 2024-02-19T21:51:57.929 INFO:teuthology.packaging:tag: None 2024-02-19T21:51:57.929 INFO:teuthology.packaging:branch: pacific-release 2024-02-19T21:51:57.929 INFO:teuthology.packaging:sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T21:51:57.930 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific-release 2024-02-19T21:51:58.153 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-548.g81bd20d6 2024-02-19T21:51:58.154 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-19T21:51:58.161 INFO:teuthology.task.internal:no buildpackages task found 2024-02-19T21:51:58.161 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-19T21:51:58.167 INFO:teuthology.task.internal:Saving configuration 2024-02-19T21:51:58.186 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-19T21:51:58.192 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-19T21:51:58.218 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi052.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566803', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 21:46:04.763280', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK3gdrbO5novp6qfB1dH4FvuFEW8gqbmsgtUnuKo6FK5q5Z2YsmKPlBawQApgt+K1xXvoIYa9OXRPIL808vgEs4='} 2024-02-19T21:51:58.235 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi088.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566803', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 21:46:04.764852', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJsfHsnFkPbhk/aA3BZaiEoJv0bulu+9B1Y4Z9juUllAkaC2GW4G+K0qS9Zxw44kth7QvmHbxwlmvRgZ3K2bAes='} 2024-02-19T21:51:58.235 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-19T21:51:58.245 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-02-19T21:51:58.245 INFO:teuthology.task.internal:roles: ubuntu@smithi088.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-02-19T21:51:58.245 INFO:teuthology.run_tasks:Running task console_log... 2024-02-19T21:51:58.314 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f27259d8f70>, signals=[15]) 2024-02-19T21:51:58.314 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-19T21:51:58.320 INFO:teuthology.task.internal:Opening connections... 2024-02-19T21:51:58.320 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi052.front.sepia.ceph.com 2024-02-19T21:51:58.322 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:51:58.399 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi088.front.sepia.ceph.com 2024-02-19T21:51:58.400 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:51:58.470 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-19T21:51:58.476 DEBUG:teuthology.orchestra.run.smithi052:> uname -m 2024-02-19T21:51:58.499 INFO:teuthology.orchestra.run.smithi052.stdout:x86_64 2024-02-19T21:51:58.500 DEBUG:teuthology.orchestra.run.smithi052:> cat /etc/os-release 2024-02-19T21:51:58.561 INFO:teuthology.orchestra.run.smithi052.stdout:NAME="CentOS Stream" 2024-02-19T21:51:58.562 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION="8" 2024-02-19T21:51:58.562 INFO:teuthology.orchestra.run.smithi052.stdout:ID="centos" 2024-02-19T21:51:58.562 INFO:teuthology.orchestra.run.smithi052.stdout:ID_LIKE="rhel fedora" 2024-02-19T21:51:58.562 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION_ID="8" 2024-02-19T21:51:58.562 INFO:teuthology.orchestra.run.smithi052.stdout:PLATFORM_ID="platform:el8" 2024-02-19T21:51:58.562 INFO:teuthology.orchestra.run.smithi052.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-19T21:51:58.562 INFO:teuthology.orchestra.run.smithi052.stdout:ANSI_COLOR="0;31" 2024-02-19T21:51:58.562 INFO:teuthology.orchestra.run.smithi052.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-19T21:51:58.562 INFO:teuthology.orchestra.run.smithi052.stdout:HOME_URL="https://centos.org/" 2024-02-19T21:51:58.562 INFO:teuthology.orchestra.run.smithi052.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-19T21:51:58.562 INFO:teuthology.orchestra.run.smithi052.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-19T21:51:58.562 INFO:teuthology.orchestra.run.smithi052.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-19T21:51:58.563 INFO:teuthology.lock.ops:Updating smithi052.front.sepia.ceph.com on lock server 2024-02-19T21:51:58.583 DEBUG:teuthology.orchestra.run.smithi088:> uname -m 2024-02-19T21:51:58.605 INFO:teuthology.orchestra.run.smithi088.stdout:x86_64 2024-02-19T21:51:58.605 DEBUG:teuthology.orchestra.run.smithi088:> cat /etc/os-release 2024-02-19T21:51:58.665 INFO:teuthology.orchestra.run.smithi088.stdout:NAME="CentOS Stream" 2024-02-19T21:51:58.666 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION="8" 2024-02-19T21:51:58.666 INFO:teuthology.orchestra.run.smithi088.stdout:ID="centos" 2024-02-19T21:51:58.666 INFO:teuthology.orchestra.run.smithi088.stdout:ID_LIKE="rhel fedora" 2024-02-19T21:51:58.666 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION_ID="8" 2024-02-19T21:51:58.666 INFO:teuthology.orchestra.run.smithi088.stdout:PLATFORM_ID="platform:el8" 2024-02-19T21:51:58.666 INFO:teuthology.orchestra.run.smithi088.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-19T21:51:58.666 INFO:teuthology.orchestra.run.smithi088.stdout:ANSI_COLOR="0;31" 2024-02-19T21:51:58.666 INFO:teuthology.orchestra.run.smithi088.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-19T21:51:58.666 INFO:teuthology.orchestra.run.smithi088.stdout:HOME_URL="https://centos.org/" 2024-02-19T21:51:58.666 INFO:teuthology.orchestra.run.smithi088.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-19T21:51:58.666 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-19T21:51:58.666 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-19T21:51:58.667 INFO:teuthology.lock.ops:Updating smithi088.front.sepia.ceph.com on lock server 2024-02-19T21:51:58.691 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-19T21:51:58.700 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-19T21:51:58.725 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-19T21:51:58.725 DEBUG:teuthology.orchestra.run.smithi052:> test '!' -e /home/ubuntu/cephtest 2024-02-19T21:51:58.728 DEBUG:teuthology.orchestra.run.smithi088:> test '!' -e /home/ubuntu/cephtest 2024-02-19T21:51:58.748 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-19T21:51:58.754 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-19T21:51:58.755 DEBUG:teuthology.orchestra.run.smithi052:> test -z $(ls -A /var/lib/ceph) 2024-02-19T21:51:58.791 DEBUG:teuthology.orchestra.run.smithi088:> test -z $(ls -A /var/lib/ceph) 2024-02-19T21:51:58.823 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-19T21:51:58.878 INFO:teuthology.run_tasks:Running task kernel... 2024-02-19T21:51:58.894 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-19T21:51:58.894 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-19T21:51:58.895 DEBUG:teuthology.orchestra.run.smithi052:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-19T21:51:58.895 DEBUG:teuthology.orchestra.run.smithi088:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-19T21:51:58.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T21:51:58.916 DEBUG:teuthology.orchestra.run.smithi088:> uname -r 2024-02-19T21:51:58.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T21:51:58.919 DEBUG:teuthology.orchestra.run.smithi052:> uname -r 2024-02-19T21:51:58.978 INFO:teuthology.orchestra.run.smithi088.stdout:4.18.0-489.el8.x86_64 2024-02-19T21:51:58.978 INFO:teuthology.task.kernel:Running kernel on smithi088: 4.18.0-489.el8.x86_64 2024-02-19T21:51:58.978 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum install -y kernel 2024-02-19T21:51:58.981 INFO:teuthology.orchestra.run.smithi052.stdout:4.18.0-489.el8.x86_64 2024-02-19T21:51:58.981 INFO:teuthology.task.kernel:Running kernel on smithi052: 4.18.0-489.el8.x86_64 2024-02-19T21:51:58.981 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum install -y kernel 2024-02-19T21:52:10.882 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 8 - AppStream 2.9 MB/s | 27 MB 00:09 2024-02-19T21:52:11.035 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 8 - AppStream 2.9 MB/s | 27 MB 00:09 2024-02-19T21:52:16.551 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 8 - BaseOS 14 MB/s | 10 MB 00:00 2024-02-19T21:52:16.909 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 8 - BaseOS 12 MB/s | 10 MB 00:00 2024-02-19T21:52:18.252 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 8 - Extras 166 kB/s | 18 kB 00:00 2024-02-19T21:52:18.486 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 8 - Extras common packages 55 kB/s | 6.9 kB 00:00 2024-02-19T21:52:18.712 INFO:teuthology.orchestra.run.smithi088.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2024-02-19T21:52:18.760 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 8 - Extras 112 kB/s | 18 kB 00:00 2024-02-19T21:52:19.018 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 8 - Extras common packages 49 kB/s | 6.9 kB 00:00 2024-02-19T21:52:19.253 INFO:teuthology.orchestra.run.smithi052.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2024-02-19T21:52:20.998 INFO:teuthology.orchestra.run.smithi052.stdout:Extra Packages for Enterprise Linux 9.8 MB/s | 16 MB 00:01 2024-02-19T21:52:22.990 INFO:teuthology.orchestra.run.smithi088.stdout:Extra Packages for Enterprise Linux 3.9 MB/s | 16 MB 00:04 2024-02-19T21:52:24.955 INFO:teuthology.orchestra.run.smithi052.stdout:lab-extras 534 kB/s | 24 kB 00:00 2024-02-19T21:52:26.854 INFO:teuthology.orchestra.run.smithi088.stdout:lab-extras 393 kB/s | 24 kB 00:00 2024-02-19T21:52:28.076 INFO:teuthology.orchestra.run.smithi052.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-19T21:52:28.076 INFO:teuthology.orchestra.run.smithi052.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-19T21:52:28.202 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-02-19T21:52:28.203 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T21:52:28.203 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2024-02-19T21:52:28.203 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T21:52:28.203 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-02-19T21:52:28.203 INFO:teuthology.orchestra.run.smithi052.stdout: kernel x86_64 4.18.0-539.el8 baseos 10 M 2024-02-19T21:52:28.203 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-02-19T21:52:28.203 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-core x86_64 4.18.0-539.el8 baseos 43 M 2024-02-19T21:52:28.203 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-modules x86_64 4.18.0-539.el8 baseos 36 M 2024-02-19T21:52:28.203 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T21:52:28.203 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-02-19T21:52:28.203 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T21:52:28.203 INFO:teuthology.orchestra.run.smithi052.stdout:Install 3 Packages 2024-02-19T21:52:28.203 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T21:52:28.204 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 90 M 2024-02-19T21:52:28.204 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 97 M 2024-02-19T21:52:28.204 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-02-19T21:52:28.981 INFO:teuthology.orchestra.run.smithi052.stdout:(1/3): kernel-4.18.0-539.el8.x86_64.rpm 14 MB/s | 10 MB 00:00 2024-02-19T21:52:29.784 INFO:teuthology.orchestra.run.smithi088.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-19T21:52:29.784 INFO:teuthology.orchestra.run.smithi088.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-19T21:52:29.902 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-02-19T21:52:29.903 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T21:52:29.903 INFO:teuthology.orchestra.run.smithi088.stdout: Package Architecture Version Repository Size 2024-02-19T21:52:29.903 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T21:52:29.903 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-02-19T21:52:29.903 INFO:teuthology.orchestra.run.smithi088.stdout: kernel x86_64 4.18.0-539.el8 baseos 10 M 2024-02-19T21:52:29.903 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-02-19T21:52:29.903 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-core x86_64 4.18.0-539.el8 baseos 43 M 2024-02-19T21:52:29.903 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-modules x86_64 4.18.0-539.el8 baseos 36 M 2024-02-19T21:52:29.903 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T21:52:29.904 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-02-19T21:52:29.904 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T21:52:29.904 INFO:teuthology.orchestra.run.smithi088.stdout:Install 3 Packages 2024-02-19T21:52:29.904 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T21:52:29.904 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 90 M 2024-02-19T21:52:29.904 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 97 M 2024-02-19T21:52:29.904 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-02-19T21:52:30.039 INFO:teuthology.orchestra.run.smithi052.stdout:(2/3): kernel-modules-4.18.0-539.el8.x86_64.rpm 20 MB/s | 36 MB 00:01 2024-02-19T21:52:30.540 INFO:teuthology.orchestra.run.smithi052.stdout:(3/3): kernel-core-4.18.0-539.el8.x86_64.rpm 19 MB/s | 43 MB 00:02 2024-02-19T21:52:30.540 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:52:30.540 INFO:teuthology.orchestra.run.smithi052.stdout:Total 38 MB/s | 90 MB 00:02 2024-02-19T21:52:30.755 INFO:teuthology.orchestra.run.smithi088.stdout:(1/3): kernel-4.18.0-539.el8.x86_64.rpm 13 MB/s | 10 MB 00:00 2024-02-19T21:52:31.255 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-02-19T21:52:31.718 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-02-19T21:52:31.718 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-02-19T21:52:31.980 INFO:teuthology.orchestra.run.smithi088.stdout:(2/3): kernel-modules-4.18.0-539.el8.x86_64.rpm 18 MB/s | 36 MB 00:02 2024-02-19T21:52:32.405 INFO:teuthology.orchestra.run.smithi088.stdout:(3/3): kernel-core-4.18.0-539.el8.x86_64.rpm 18 MB/s | 43 MB 00:02 2024-02-19T21:52:32.406 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:52:32.406 INFO:teuthology.orchestra.run.smithi088.stdout:Total 36 MB/s | 90 MB 00:02 2024-02-19T21:52:33.031 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-02-19T21:52:33.051 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-02-19T21:52:33.084 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-02-19T21:52:33.511 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-02-19T21:52:33.511 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-02-19T21:52:34.755 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-02-19T21:52:34.770 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-02-19T21:52:35.962 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-02-19T21:52:36.467 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-19T21:52:37.558 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-02-19T21:52:38.035 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-19T21:52:40.247 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-19T21:52:40.649 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-19T21:52:41.597 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-19T21:52:41.976 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-19T21:52:44.325 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-19T21:52:44.626 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:52:45.418 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-19T21:52:45.911 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : kernel-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:53:07.764 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:53:07.786 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:53:07.946 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:53:07.971 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:53:08.127 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:53:08.127 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : kernel-4.18.0-539.el8.x86_64 1/3 2024-02-19T21:53:08.127 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : kernel-core-4.18.0-539.el8.x86_64 2/3 2024-02-19T21:53:08.333 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:53:08.334 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-4.18.0-539.el8.x86_64 1/3 2024-02-19T21:53:08.334 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-core-4.18.0-539.el8.x86_64 2/3 2024-02-19T21:53:08.591 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:53:08.591 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T21:53:08.591 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-02-19T21:53:08.591 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-4.18.0-539.el8.x86_64 kernel-core-4.18.0-539.el8.x86_64 2024-02-19T21:53:08.592 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-modules-4.18.0-539.el8.x86_64 2024-02-19T21:53:08.592 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T21:53:08.592 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-02-19T21:53:08.765 DEBUG:teuthology.orchestra.run.smithi088:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T21:53:08.841 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:53:08.841 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T21:53:08.842 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-02-19T21:53:08.842 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-4.18.0-539.el8.x86_64 kernel-core-4.18.0-539.el8.x86_64 2024-02-19T21:53:08.842 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-modules-4.18.0-539.el8.x86_64 2024-02-19T21:53:08.842 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T21:53:08.842 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-02-19T21:53:08.985 INFO:teuthology.orchestra.run.smithi088.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-19T21:53:08.985 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-19T21:53:08.985 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-539.el8.x86_64 2024-02-19T21:53:08.985 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-19T21:53:08.986 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-19T21:53:08.986 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-02-19T21:53:08.986 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi088.front.sepia.ceph.com, path=None, version=distro) 2024-02-19T21:53:08.986 DEBUG:teuthology.orchestra.run.smithi088:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T21:53:09.196 INFO:teuthology.orchestra.run.smithi088.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-19T21:53:09.196 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-19T21:53:09.196 DEBUG:teuthology.orchestra.run.smithi088:> sudo rpm -qi grub2-tools 2024-02-19T21:53:09.229 INFO:teuthology.orchestra.run.smithi088.stdout:Name : grub2-tools 2024-02-19T21:53:09.229 INFO:teuthology.orchestra.run.smithi088.stdout:Epoch : 1 2024-02-19T21:53:09.229 INFO:teuthology.orchestra.run.smithi088.stdout:Version : 2.02 2024-02-19T21:53:09.230 INFO:teuthology.orchestra.run.smithi088.stdout:Release : 129.el8 2024-02-19T21:53:09.230 INFO:teuthology.orchestra.run.smithi088.stdout:Architecture: x86_64 2024-02-19T21:53:09.230 INFO:teuthology.orchestra.run.smithi088.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-19T21:53:09.230 INFO:teuthology.orchestra.run.smithi088.stdout:Group : System Environment/Base 2024-02-19T21:53:09.230 INFO:teuthology.orchestra.run.smithi088.stdout:Size : 9265586 2024-02-19T21:53:09.230 INFO:teuthology.orchestra.run.smithi088.stdout:License : GPLv3+ 2024-02-19T21:53:09.230 INFO:teuthology.orchestra.run.smithi088.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-19T21:53:09.230 INFO:teuthology.orchestra.run.smithi088.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-19T21:53:09.230 INFO:teuthology.orchestra.run.smithi088.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-19T21:53:09.230 INFO:teuthology.orchestra.run.smithi088.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-19T21:53:09.230 INFO:teuthology.orchestra.run.smithi088.stdout:Relocations : (not relocatable) 2024-02-19T21:53:09.230 INFO:teuthology.orchestra.run.smithi088.stdout:Packager : CentOS BuildSystem 2024-02-19T21:53:09.230 INFO:teuthology.orchestra.run.smithi088.stdout:Vendor : CentOS 2024-02-19T21:53:09.230 INFO:teuthology.orchestra.run.smithi088.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-19T21:53:09.230 INFO:teuthology.orchestra.run.smithi088.stdout:Summary : Support tools for GRUB. 2024-02-19T21:53:09.231 INFO:teuthology.orchestra.run.smithi088.stdout:Description : 2024-02-19T21:53:09.231 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T21:53:09.231 INFO:teuthology.orchestra.run.smithi088.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-19T21:53:09.231 INFO:teuthology.orchestra.run.smithi088.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-19T21:53:09.231 INFO:teuthology.orchestra.run.smithi088.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-19T21:53:09.231 INFO:teuthology.orchestra.run.smithi088.stdout:hardware devices. 2024-02-19T21:53:09.231 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T21:53:09.231 INFO:teuthology.orchestra.run.smithi088.stdout:This subpackage provides tools for support of all platforms. 2024-02-19T21:53:09.232 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-19T21:53:09.232 INFO:teuthology.task.kernel:Updating grub on smithi088 to boot 4.18.0-539.el8.x86_64 2024-02-19T21:53:09.232 DEBUG:teuthology.orchestra.run.smithi088:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-19T21:53:09.922 DEBUG:teuthology.orchestra.run.smithi052:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T21:53:10.168 INFO:teuthology.orchestra.run.smithi052.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-19T21:53:10.168 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-19T21:53:10.168 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-539.el8.x86_64 2024-02-19T21:53:10.168 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-19T21:53:10.169 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-19T21:53:10.169 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-02-19T21:53:10.169 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi052.front.sepia.ceph.com, path=None, version=distro) 2024-02-19T21:53:10.169 DEBUG:teuthology.orchestra.run.smithi052:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T21:53:10.387 INFO:teuthology.orchestra.run.smithi052.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-19T21:53:10.388 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-19T21:53:10.388 DEBUG:teuthology.orchestra.run.smithi052:> sudo rpm -qi grub2-tools 2024-02-19T21:53:10.422 INFO:teuthology.orchestra.run.smithi052.stdout:Name : grub2-tools 2024-02-19T21:53:10.422 INFO:teuthology.orchestra.run.smithi052.stdout:Epoch : 1 2024-02-19T21:53:10.422 INFO:teuthology.orchestra.run.smithi052.stdout:Version : 2.02 2024-02-19T21:53:10.422 INFO:teuthology.orchestra.run.smithi052.stdout:Release : 129.el8 2024-02-19T21:53:10.422 INFO:teuthology.orchestra.run.smithi052.stdout:Architecture: x86_64 2024-02-19T21:53:10.422 INFO:teuthology.orchestra.run.smithi052.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-19T21:53:10.422 INFO:teuthology.orchestra.run.smithi052.stdout:Group : System Environment/Base 2024-02-19T21:53:10.422 INFO:teuthology.orchestra.run.smithi052.stdout:Size : 9265586 2024-02-19T21:53:10.422 INFO:teuthology.orchestra.run.smithi052.stdout:License : GPLv3+ 2024-02-19T21:53:10.422 INFO:teuthology.orchestra.run.smithi052.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-19T21:53:10.422 INFO:teuthology.orchestra.run.smithi052.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-19T21:53:10.422 INFO:teuthology.orchestra.run.smithi052.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-19T21:53:10.422 INFO:teuthology.orchestra.run.smithi052.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-19T21:53:10.422 INFO:teuthology.orchestra.run.smithi052.stdout:Relocations : (not relocatable) 2024-02-19T21:53:10.423 INFO:teuthology.orchestra.run.smithi052.stdout:Packager : CentOS BuildSystem 2024-02-19T21:53:10.423 INFO:teuthology.orchestra.run.smithi052.stdout:Vendor : CentOS 2024-02-19T21:53:10.423 INFO:teuthology.orchestra.run.smithi052.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-19T21:53:10.423 INFO:teuthology.orchestra.run.smithi052.stdout:Summary : Support tools for GRUB. 2024-02-19T21:53:10.423 INFO:teuthology.orchestra.run.smithi052.stdout:Description : 2024-02-19T21:53:10.423 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T21:53:10.423 INFO:teuthology.orchestra.run.smithi052.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-19T21:53:10.423 INFO:teuthology.orchestra.run.smithi052.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-19T21:53:10.423 INFO:teuthology.orchestra.run.smithi052.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-19T21:53:10.423 INFO:teuthology.orchestra.run.smithi052.stdout:hardware devices. 2024-02-19T21:53:10.423 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T21:53:10.423 INFO:teuthology.orchestra.run.smithi052.stdout:This subpackage provides tools for support of all platforms. 2024-02-19T21:53:10.424 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-19T21:53:10.424 INFO:teuthology.task.kernel:Updating grub on smithi052 to boot 4.18.0-539.el8.x86_64 2024-02-19T21:53:10.424 DEBUG:teuthology.orchestra.run.smithi052:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-19T21:53:10.773 INFO:teuthology.orchestra.run.smithi088.stderr:Generating grub configuration file ... 2024-02-19T21:53:11.490 INFO:teuthology.orchestra.run.smithi052.stderr:Generating grub configuration file ... 2024-02-19T21:53:12.622 INFO:teuthology.orchestra.run.smithi088.stderr:done 2024-02-19T21:53:12.623 DEBUG:teuthology.orchestra.run.smithi088:> mktemp 2024-02-19T21:53:12.647 INFO:teuthology.orchestra.run.smithi088.stdout:/tmp/tmp.mMOkAdUcbf 2024-02-19T21:53:12.647 DEBUG:teuthology.orchestra.run.smithi088:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.mMOkAdUcbf 2024-02-19T21:53:12.738 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0666 /tmp/tmp.mMOkAdUcbf 2024-02-19T21:53:12.841 DEBUG:teuthology.orchestra.remote:smithi088:/tmp/tmp.mMOkAdUcbf is 6KB 2024-02-19T21:53:12.889 DEBUG:teuthology.orchestra.run.smithi088:> rm -fr /tmp/tmp.mMOkAdUcbf 2024-02-19T21:53:12.906 DEBUG:teuthology.orchestra.run.smithi088:> sudo /bin/ls /boot/loader/entries || true 2024-02-19T21:53:12.980 INFO:teuthology.orchestra.run.smithi088.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-02-19T21:53:12.980 INFO:teuthology.orchestra.run.smithi088.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-02-19T21:53:12.980 INFO:teuthology.orchestra.run.smithi088.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-02-19T21:53:12.980 INFO:teuthology.orchestra.run.smithi088.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-539.el8.x86_64.conf 2024-02-19T21:53:12.982 DEBUG:teuthology.orchestra.run.smithi088:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-539.el8.x86_64 2024-02-19T21:53:13.128 DEBUG:teuthology.orchestra.run.smithi088:> sudo shutdown -r now 2024-02-19T21:53:13.170 INFO:teuthology.orchestra.run.smithi052.stderr:done 2024-02-19T21:53:13.171 DEBUG:teuthology.orchestra.run.smithi052:> mktemp 2024-02-19T21:53:13.193 INFO:teuthology.orchestra.run.smithi052.stdout:/tmp/tmp.eEHQB5qb8w 2024-02-19T21:53:13.193 DEBUG:teuthology.orchestra.run.smithi052:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.eEHQB5qb8w 2024-02-19T21:53:13.283 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0666 /tmp/tmp.eEHQB5qb8w 2024-02-19T21:53:13.383 DEBUG:teuthology.orchestra.remote:smithi052:/tmp/tmp.eEHQB5qb8w is 6KB 2024-02-19T21:53:13.431 DEBUG:teuthology.orchestra.run.smithi052:> rm -fr /tmp/tmp.eEHQB5qb8w 2024-02-19T21:53:13.449 DEBUG:teuthology.orchestra.run.smithi052:> sudo /bin/ls /boot/loader/entries || true 2024-02-19T21:53:13.559 INFO:teuthology.orchestra.run.smithi052.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-02-19T21:53:13.560 INFO:teuthology.orchestra.run.smithi052.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-02-19T21:53:13.560 INFO:teuthology.orchestra.run.smithi052.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-02-19T21:53:13.560 INFO:teuthology.orchestra.run.smithi052.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-539.el8.x86_64.conf 2024-02-19T21:53:13.561 DEBUG:teuthology.orchestra.run.smithi052:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-539.el8.x86_64 2024-02-19T21:53:13.719 DEBUG:teuthology.orchestra.run.smithi052:> sudo shutdown -r now 2024-02-19T21:53:43.134 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-02-19T21:53:43.134 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-02-19T21:53:43.135 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:53:43.723 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-02-19T21:53:43.723 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-02-19T21:53:43.723 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:54:01.660 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2024-02-19T21:54:02.172 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-02-19T21:54:10.670 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-02-19T21:54:10.671 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:54:11.173 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-02-19T21:54:11.174 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:54:13.728 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2024-02-19T21:54:25.740 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-02-19T21:54:25.741 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:54:28.800 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2024-02-19T21:54:43.815 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-02-19T21:54:43.816 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:55:02.332 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2024-02-19T21:55:11.184 DEBUG:teuthology.orchestra.remote:timed out 2024-02-19T21:55:20.342 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-02-19T21:55:20.343 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:55:20.652 DEBUG:teuthology.orchestra.run.smithi088:> true 2024-02-19T21:55:20.955 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-02-19T21:55:20.956 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-539.el8.x86_64"... 2024-02-19T21:55:20.956 DEBUG:teuthology.orchestra.run.smithi088:> uname -r 2024-02-19T21:55:20.975 INFO:teuthology.orchestra.run.smithi088.stdout:4.18.0-539.el8.x86_64 2024-02-19T21:55:20.976 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-539.el8.x86_64 vs 4.18.0-539.el8.x86_64 2024-02-19T21:55:20.976 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-19T21:55:20.976 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-19T21:55:21.976 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-02-19T21:55:21.977 DEBUG:teuthology.orchestra.run.smithi088:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-19T21:55:22.035 INFO:teuthology.orchestra.run.smithi088.stdout:ttyS1 2024-02-19T21:55:22.048 DEBUG:teuthology.parallel:result is None 2024-02-19T21:55:23.186 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-02-19T21:55:23.186 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:55:23.498 DEBUG:teuthology.orchestra.run.smithi052:> true 2024-02-19T21:55:23.801 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-02-19T21:55:23.801 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-539.el8.x86_64"... 2024-02-19T21:55:23.801 DEBUG:teuthology.orchestra.run.smithi052:> uname -r 2024-02-19T21:55:23.821 INFO:teuthology.orchestra.run.smithi052.stdout:4.18.0-539.el8.x86_64 2024-02-19T21:55:23.821 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-539.el8.x86_64 vs 4.18.0-539.el8.x86_64 2024-02-19T21:55:23.821 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-19T21:55:23.821 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-19T21:55:24.822 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-02-19T21:55:24.823 DEBUG:teuthology.orchestra.run.smithi052:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-19T21:55:24.884 INFO:teuthology.orchestra.run.smithi052.stdout:ttyS1 2024-02-19T21:55:24.896 DEBUG:teuthology.parallel:result is None 2024-02-19T21:55:24.896 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-19T21:55:24.903 INFO:teuthology.task.internal:Creating test directory... 2024-02-19T21:55:24.904 DEBUG:teuthology.orchestra.run.smithi052:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-19T21:55:24.906 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-19T21:55:24.927 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-19T21:55:24.933 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-19T21:55:24.943 INFO:teuthology.task.internal:Creating archive directory... 2024-02-19T21:55:24.944 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-19T21:55:24.967 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-19T21:55:24.999 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-19T21:55:25.005 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-19T21:55:25.006 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-19T21:55:25.037 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-19T21:55:25.072 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T21:55:25.074 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T21:55:25.083 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T21:55:25.085 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T21:55:25.086 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-19T21:55:25.093 INFO:teuthology.task.internal:Configuring sudo... 2024-02-19T21:55:25.093 DEBUG:teuthology.orchestra.run.smithi052:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-19T21:55:25.129 DEBUG:teuthology.orchestra.run.smithi088:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-19T21:55:25.158 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-19T21:55:25.168 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-19T21:55:25.168 DEBUG:teuthology.orchestra.run.smithi052:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-19T21:55:25.199 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-19T21:55:25.218 DEBUG:teuthology.orchestra.run.smithi052:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T21:55:25.303 DEBUG:teuthology.orchestra.run.smithi052:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T21:55:25.359 DEBUG:teuthology.orchestra.run.smithi052:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T21:55:25.446 DEBUG:teuthology.orchestra.run.smithi052:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T21:55:25.488 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-02-19T21:55:25.489 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-19T21:55:25.556 DEBUG:teuthology.orchestra.run.smithi088:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T21:55:25.602 DEBUG:teuthology.orchestra.run.smithi088:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T21:55:25.660 DEBUG:teuthology.orchestra.run.smithi088:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T21:55:25.746 DEBUG:teuthology.orchestra.run.smithi088:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T21:55:25.788 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-02-19T21:55:25.789 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-19T21:55:25.856 DEBUG:teuthology.orchestra.run.smithi052:> sudo service rsyslog restart 2024-02-19T21:55:25.858 DEBUG:teuthology.orchestra.run.smithi088:> sudo service rsyslog restart 2024-02-19T21:55:25.911 INFO:teuthology.orchestra.run.smithi052.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-19T21:55:25.958 INFO:teuthology.orchestra.run.smithi088.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-19T21:55:26.432 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-19T21:55:26.444 INFO:teuthology.task.internal:Starting timer... 2024-02-19T21:55:26.444 INFO:teuthology.run_tasks:Running task pcp... 2024-02-19T21:55:26.459 INFO:teuthology.run_tasks:Running task selinux... 2024-02-19T21:55:26.472 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-02-19T21:55:26.473 DEBUG:teuthology.orchestra.run.smithi052:> sudo service auditd rotate 2024-02-19T21:55:26.547 INFO:teuthology.orchestra.run.smithi052.stdout:Rotating logs: 2024-02-19T21:55:26.548 DEBUG:teuthology.orchestra.run.smithi088:> sudo service auditd rotate 2024-02-19T21:55:26.619 INFO:teuthology.orchestra.run.smithi088.stdout:Rotating logs: 2024-02-19T21:55:26.621 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-19T21:55:26.621 DEBUG:teuthology.orchestra.run.smithi052:> /usr/sbin/getenforce 2024-02-19T21:55:26.648 INFO:teuthology.orchestra.run.smithi052.stdout:Permissive 2024-02-19T21:55:26.648 DEBUG:teuthology.orchestra.run.smithi088:> /usr/sbin/getenforce 2024-02-19T21:55:26.668 INFO:teuthology.orchestra.run.smithi088.stdout:Permissive 2024-02-19T21:55:26.669 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi052.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi088.front.sepia.ceph.com': 'permissive'} 2024-02-19T21:55:26.669 DEBUG:teuthology.orchestra.run.smithi052:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-19T21:55:26.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T21:55:26.718 DEBUG:teuthology.orchestra.run.smithi088:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-19T21:55:26.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T21:55:26.745 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-19T21:55:26.745 DEBUG:teuthology.orchestra.run.smithi052:> sudo /usr/sbin/setenforce permissive 2024-02-19T21:55:26.788 DEBUG:teuthology.orchestra.run.smithi088:> sudo /usr/sbin/setenforce permissive 2024-02-19T21:55:26.814 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-19T21:55:26.825 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-19T21:55:26.840 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-19T21:55:26.841 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi052.front.sepia.ceph.com,smithi088.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-19T22:01:40.532 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi052.front.sepia.ceph.com'), Remote(name='ubuntu@smithi088.front.sepia.ceph.com')] 2024-02-19T22:01:40.533 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-02-19T22:01:40.535 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T22:01:40.613 DEBUG:teuthology.orchestra.run.smithi052:> true 2024-02-19T22:01:40.684 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-02-19T22:01:40.684 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-02-19T22:01:40.685 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T22:01:40.758 DEBUG:teuthology.orchestra.run.smithi088:> true 2024-02-19T22:01:40.827 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-02-19T22:01:40.828 INFO:teuthology.run_tasks:Running task clock... 2024-02-19T22:01:40.838 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-19T22:01:40.839 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-19T22:01:40.839 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-19T22:01:40.841 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-19T22:01:40.841 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-19T22:01:40.873 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-19T22:01:40.888 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-19T22:01:40.900 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-19T22:01:40.914 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-19T22:01:40.960 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: ntpd: command not found 2024-02-19T22:01:40.971 INFO:teuthology.orchestra.run.smithi052.stdout:506 Cannot talk to daemon 2024-02-19T22:01:40.978 INFO:teuthology.orchestra.run.smithi088.stderr:sudo: ntpd: command not found 2024-02-19T22:01:40.986 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-19T22:01:40.989 INFO:teuthology.orchestra.run.smithi088.stdout:506 Cannot talk to daemon 2024-02-19T22:01:40.999 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-19T22:01:41.002 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-19T22:01:41.015 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-19T22:01:41.072 INFO:teuthology.orchestra.run.smithi052.stderr:bash: ntpq: command not found 2024-02-19T22:01:41.076 INFO:teuthology.orchestra.run.smithi052.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-19T22:01:41.077 INFO:teuthology.orchestra.run.smithi052.stdout:=============================================================================== 2024-02-19T22:01:41.077 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T22:01:41.077 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T22:01:41.077 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T22:01:41.077 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T22:01:41.079 INFO:teuthology.orchestra.run.smithi088.stderr:bash: ntpq: command not found 2024-02-19T22:01:41.082 INFO:teuthology.orchestra.run.smithi088.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-19T22:01:41.082 INFO:teuthology.orchestra.run.smithi088.stdout:=============================================================================== 2024-02-19T22:01:41.082 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T22:01:41.082 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T22:01:41.082 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T22:01:41.083 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T22:01:41.083 INFO:teuthology.run_tasks:Running task pexec... 2024-02-19T22:01:41.093 INFO:teuthology.task.pexec:Executing custom commands... 2024-02-19T22:01:41.093 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi052.front.sepia.ceph.com 2024-02-19T22:01:41.094 DEBUG:teuthology.orchestra.run.smithi052:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-19T22:01:41.094 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi088.front.sepia.ceph.com 2024-02-19T22:01:41.094 DEBUG:teuthology.orchestra.run.smithi088:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-19T22:01:41.904 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:48 ago on Mon 19 Feb 2024 09:59:53 PM UTC. 2024-02-19T22:01:41.930 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:39 ago on Mon 19 Feb 2024 10:00:02 PM UTC. 2024-02-19T22:01:43.925 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-02-19T22:01:43.926 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:01:43.926 INFO:teuthology.orchestra.run.smithi088.stdout: Package Architecture Version Repository Size 2024-02-19T22:01:43.926 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:01:43.926 INFO:teuthology.orchestra.run.smithi088.stdout:Resetting modules: 2024-02-19T22:01:43.926 INFO:teuthology.orchestra.run.smithi088.stdout: container-tools 2024-02-19T22:01:43.926 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:01:43.927 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-02-19T22:01:43.927 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:01:43.927 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:01:44.120 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-02-19T22:01:44.121 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:01:44.121 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2024-02-19T22:01:44.121 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:01:44.121 INFO:teuthology.orchestra.run.smithi052.stdout:Resetting modules: 2024-02-19T22:01:44.121 INFO:teuthology.orchestra.run.smithi052.stdout: container-tools 2024-02-19T22:01:44.121 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:01:44.121 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-02-19T22:01:44.121 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:01:44.121 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:01:44.175 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-02-19T22:01:44.389 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-02-19T22:01:45.028 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:51 ago on Mon 19 Feb 2024 09:59:53 PM UTC. 2024-02-19T22:01:45.323 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:43 ago on Mon 19 Feb 2024 10:00:02 PM UTC. 2024-02-19T22:01:47.065 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-02-19T22:01:47.101 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:01:47.101 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-02-19T22:01:47.101 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:01:47.101 INFO:teuthology.orchestra.run.smithi088.stdout:Installing group/module packages: 2024-02-19T22:01:47.101 INFO:teuthology.orchestra.run.smithi088.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-02-19T22:01:47.101 INFO:teuthology.orchestra.run.smithi088.stdout: buildah x86_64 2:1.33.5-1.module_el8+885+7da147f3 appstream 9.7 M 2024-02-19T22:01:47.101 INFO:teuthology.orchestra.run.smithi088.stdout: cockpit-podman noarch 83-1.module_el8+847+7863d4e6 appstream 688 k 2024-02-19T22:01:47.101 INFO:teuthology.orchestra.run.smithi088.stdout: crun x86_64 1.14-1.module_el8+871+552be958 appstream 257 k 2024-02-19T22:01:47.102 INFO:teuthology.orchestra.run.smithi088.stdout: netavark x86_64 2:1.10.2-1.module_el8+885+7da147f3 appstream 4.3 M 2024-02-19T22:01:47.102 INFO:teuthology.orchestra.run.smithi088.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-02-19T22:01:47.102 INFO:teuthology.orchestra.run.smithi088.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-02-19T22:01:47.102 INFO:teuthology.orchestra.run.smithi088.stdout: toolbox x86_64 0.0.99.5-1.module_el8+847+7863d4e6 appstream 2.5 M 2024-02-19T22:01:47.102 INFO:teuthology.orchestra.run.smithi088.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-02-19T22:01:47.102 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-02-19T22:01:47.102 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-02-19T22:01:47.102 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-02-19T22:01:47.102 INFO:teuthology.orchestra.run.smithi088.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-02-19T22:01:47.102 INFO:teuthology.orchestra.run.smithi088.stdout:Installing weak dependencies: 2024-02-19T22:01:47.102 INFO:teuthology.orchestra.run.smithi088.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-02-19T22:01:47.102 INFO:teuthology.orchestra.run.smithi088.stdout:Installing module profiles: 2024-02-19T22:01:47.102 INFO:teuthology.orchestra.run.smithi088.stdout: container-tools/common 2024-02-19T22:01:47.102 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:01:47.102 INFO:teuthology.orchestra.run.smithi088.stdout:Enabling module streams: 2024-02-19T22:01:47.103 INFO:teuthology.orchestra.run.smithi088.stdout: container-tools rhel8 2024-02-19T22:01:47.103 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:01:47.103 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-02-19T22:01:47.103 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:01:47.103 INFO:teuthology.orchestra.run.smithi088.stdout:Install 13 Packages 2024-02-19T22:01:47.103 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:01:47.103 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 28 M 2024-02-19T22:01:47.103 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 92 M 2024-02-19T22:01:47.103 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-02-19T22:01:47.483 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-02-19T22:01:47.521 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:01:47.521 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-02-19T22:01:47.521 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:01:47.522 INFO:teuthology.orchestra.run.smithi052.stdout:Installing group/module packages: 2024-02-19T22:01:47.522 INFO:teuthology.orchestra.run.smithi052.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-02-19T22:01:47.522 INFO:teuthology.orchestra.run.smithi052.stdout: buildah x86_64 2:1.33.5-1.module_el8+885+7da147f3 appstream 9.7 M 2024-02-19T22:01:47.522 INFO:teuthology.orchestra.run.smithi052.stdout: cockpit-podman noarch 83-1.module_el8+847+7863d4e6 appstream 688 k 2024-02-19T22:01:47.522 INFO:teuthology.orchestra.run.smithi052.stdout: crun x86_64 1.14-1.module_el8+871+552be958 appstream 257 k 2024-02-19T22:01:47.522 INFO:teuthology.orchestra.run.smithi052.stdout: netavark x86_64 2:1.10.2-1.module_el8+885+7da147f3 appstream 4.3 M 2024-02-19T22:01:47.522 INFO:teuthology.orchestra.run.smithi052.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-02-19T22:01:47.522 INFO:teuthology.orchestra.run.smithi052.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-02-19T22:01:47.522 INFO:teuthology.orchestra.run.smithi052.stdout: toolbox x86_64 0.0.99.5-1.module_el8+847+7863d4e6 appstream 2.5 M 2024-02-19T22:01:47.522 INFO:teuthology.orchestra.run.smithi052.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-02-19T22:01:47.522 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-02-19T22:01:47.522 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-02-19T22:01:47.522 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-02-19T22:01:47.522 INFO:teuthology.orchestra.run.smithi052.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-02-19T22:01:47.522 INFO:teuthology.orchestra.run.smithi052.stdout:Installing weak dependencies: 2024-02-19T22:01:47.522 INFO:teuthology.orchestra.run.smithi052.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-02-19T22:01:47.523 INFO:teuthology.orchestra.run.smithi052.stdout:Installing module profiles: 2024-02-19T22:01:47.523 INFO:teuthology.orchestra.run.smithi052.stdout: container-tools/common 2024-02-19T22:01:47.523 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:01:47.523 INFO:teuthology.orchestra.run.smithi052.stdout:Enabling module streams: 2024-02-19T22:01:47.523 INFO:teuthology.orchestra.run.smithi052.stdout: container-tools rhel8 2024-02-19T22:01:47.523 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:01:47.523 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-02-19T22:01:47.523 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:01:47.523 INFO:teuthology.orchestra.run.smithi052.stdout:Install 13 Packages 2024-02-19T22:01:47.523 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:01:47.524 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 28 M 2024-02-19T22:01:47.524 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 92 M 2024-02-19T22:01:47.524 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-02-19T22:01:47.715 INFO:teuthology.orchestra.run.smithi088.stdout:(1/13): cockpit-podman-83-1.module_el8+847+7863 1.2 MB/s | 688 kB 00:00 2024-02-19T22:01:47.815 INFO:teuthology.orchestra.run.smithi088.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 379 kB/s | 38 kB 00:00 2024-02-19T22:01:47.916 INFO:teuthology.orchestra.run.smithi088.stdout:(3/13): aardvark-dns-1.10.0-1.module_el8+871+55 1.3 MB/s | 1.0 MB 00:00 2024-02-19T22:01:48.241 INFO:teuthology.orchestra.run.smithi088.stdout:(4/13): crun-1.14-1.module_el8+871+552be958.x86 604 kB/s | 257 kB 00:00 2024-02-19T22:01:48.399 INFO:teuthology.orchestra.run.smithi088.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 1.1 MB/s | 180 kB 00:00 2024-02-19T22:01:48.508 INFO:teuthology.orchestra.run.smithi088.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 228 kB/s | 25 kB 00:00 2024-02-19T22:01:48.684 INFO:teuthology.orchestra.run.smithi088.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 536 kB/s | 94 kB 00:00 2024-02-19T22:01:48.755 INFO:teuthology.orchestra.run.smithi052.stdout:(1/13): cockpit-podman-83-1.module_el8+847+7863 659 kB/s | 688 kB 00:01 2024-02-19T22:01:48.873 INFO:teuthology.orchestra.run.smithi052.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 321 kB/s | 38 kB 00:00 2024-02-19T22:01:49.174 INFO:teuthology.orchestra.run.smithi052.stdout:(3/13): crun-1.14-1.module_el8+871+552be958.x86 856 kB/s | 257 kB 00:00 2024-02-19T22:01:49.566 INFO:teuthology.orchestra.run.smithi052.stdout:(4/13): aardvark-dns-1.10.0-1.module_el8+871+55 556 kB/s | 1.0 MB 00:01 2024-02-19T22:01:49.833 INFO:teuthology.orchestra.run.smithi052.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 676 kB/s | 180 kB 00:00 2024-02-19T22:01:49.966 INFO:teuthology.orchestra.run.smithi052.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 186 kB/s | 25 kB 00:00 2024-02-19T22:01:50.125 INFO:teuthology.orchestra.run.smithi052.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 592 kB/s | 94 kB 00:00 2024-02-19T22:01:52.408 INFO:teuthology.orchestra.run.smithi088.stdout:(8/13): netavark-1.10.2-1.module_el8+885+7da147 983 kB/s | 4.3 MB 00:04 2024-02-19T22:01:54.319 INFO:teuthology.orchestra.run.smithi052.stdout:(8/13): netavark-1.10.2-1.module_el8+885+7da147 859 kB/s | 4.3 MB 00:05 2024-02-19T22:01:54.799 INFO:teuthology.orchestra.run.smithi088.stdout:(9/13): toolbox-0.0.99.5-1.module_el8+847+7863d 1.1 MB/s | 2.5 MB 00:02 2024-02-19T22:01:54.883 INFO:teuthology.orchestra.run.smithi088.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 633 kB/s | 53 kB 00:00 2024-02-19T22:01:54.942 INFO:teuthology.orchestra.run.smithi088.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 703 kB/s | 41 kB 00:00 2024-02-19T22:01:56.175 INFO:teuthology.orchestra.run.smithi088.stdout:(12/13): skopeo-1.14.3-0.1.module_el8+885+7da14 1.2 MB/s | 8.8 MB 00:07 2024-02-19T22:01:56.316 INFO:teuthology.orchestra.run.smithi052.stdout:(9/13): toolbox-0.0.99.5-1.module_el8+847+7863d 1.3 MB/s | 2.5 MB 00:01 2024-02-19T22:01:56.333 INFO:teuthology.orchestra.run.smithi088.stdout:(13/13): buildah-1.33.5-1.module_el8+885+7da147 1.1 MB/s | 9.7 MB 00:09 2024-02-19T22:01:56.334 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:01:56.334 INFO:teuthology.orchestra.run.smithi088.stdout:Total 3.0 MB/s | 28 MB 00:09 2024-02-19T22:01:56.400 INFO:teuthology.orchestra.run.smithi052.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 634 kB/s | 53 kB 00:00 2024-02-19T22:01:56.461 INFO:teuthology.orchestra.run.smithi052.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 676 kB/s | 41 kB 00:00 2024-02-19T22:01:56.606 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-02-19T22:01:56.626 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-02-19T22:01:56.627 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-02-19T22:01:56.886 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-02-19T22:01:56.889 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-02-19T22:01:57.583 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-02-19T22:01:57.642 INFO:teuthology.orchestra.run.smithi052.stdout:(12/13): buildah-1.33.5-1.module_el8+885+7da147 996 kB/s | 9.7 MB 00:09 2024-02-19T22:01:58.524 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-02-19T22:01:58.725 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-02-19T22:01:58.725 INFO:teuthology.orchestra.run.smithi052.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 1.0 MB/s | 8.8 MB 00:08 2024-02-19T22:01:58.726 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:01:58.726 INFO:teuthology.orchestra.run.smithi052.stdout:Total 2.5 MB/s | 28 MB 00:11 2024-02-19T22:01:58.887 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-02-19T22:01:59.011 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-02-19T22:01:59.034 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-02-19T22:01:59.034 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-02-19T22:01:59.105 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-02-19T22:01:59.317 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-02-19T22:01:59.321 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-02-19T22:01:59.331 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-02-19T22:01:59.926 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-02-19T22:02:00.073 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-02-19T22:02:00.899 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 7/13 2024-02-19T22:02:01.105 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 8/13 2024-02-19T22:02:01.317 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : crun-1.14-1.module_el8+871+552be958.x86_64 9/13 2024-02-19T22:02:01.332 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-02-19T22:02:01.548 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-02-19T22:02:01.742 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-02-19T22:02:01.753 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-02-19T22:02:01.955 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-19T22:02:02.111 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-02-19T22:02:02.174 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-19T22:02:02.350 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-02-19T22:02:02.402 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-19T22:02:02.818 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-19T22:02:02.818 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-02-19T22:02:02.818 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2/13 2024-02-19T22:02:02.818 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 3/13 2024-02-19T22:02:02.818 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-19T22:02:02.818 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : crun-1.14-1.module_el8+871+552be958.x86_64 5/13 2024-02-19T22:02:02.818 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 6/13 2024-02-19T22:02:02.819 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-02-19T22:02:02.819 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-02-19T22:02:02.819 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-02-19T22:02:02.819 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-02-19T22:02:02.819 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-19T22:02:02.819 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-19T22:02:02.949 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-02-19T22:02:03.302 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-02-19T22:02:03.302 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:02:03.302 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-02-19T22:02:03.302 INFO:teuthology.orchestra.run.smithi088.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-02-19T22:02:03.302 INFO:teuthology.orchestra.run.smithi088.stdout: buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2024-02-19T22:02:03.302 INFO:teuthology.orchestra.run.smithi088.stdout: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 2024-02-19T22:02:03.302 INFO:teuthology.orchestra.run.smithi088.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-02-19T22:02:03.302 INFO:teuthology.orchestra.run.smithi088.stdout: crun-1.14-1.module_el8+871+552be958.x86_64 2024-02-19T22:02:03.302 INFO:teuthology.orchestra.run.smithi088.stdout: netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 2024-02-19T22:02:03.302 INFO:teuthology.orchestra.run.smithi088.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-02-19T22:02:03.302 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-19T22:02:03.302 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-19T22:02:03.302 INFO:teuthology.orchestra.run.smithi088.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-02-19T22:02:03.302 INFO:teuthology.orchestra.run.smithi088.stdout: toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 2024-02-19T22:02:03.302 INFO:teuthology.orchestra.run.smithi088.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-02-19T22:02:03.303 INFO:teuthology.orchestra.run.smithi088.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-19T22:02:03.303 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:02:03.303 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-02-19T22:02:03.554 DEBUG:teuthology.parallel:result is None 2024-02-19T22:02:04.095 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 7/13 2024-02-19T22:02:04.343 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 8/13 2024-02-19T22:02:04.584 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : crun-1.14-1.module_el8+871+552be958.x86_64 9/13 2024-02-19T22:02:05.081 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-02-19T22:02:05.476 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-19T22:02:05.741 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-19T22:02:05.966 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-19T22:02:06.449 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-19T22:02:06.449 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-02-19T22:02:06.449 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2/13 2024-02-19T22:02:06.449 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 3/13 2024-02-19T22:02:06.449 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-19T22:02:06.449 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : crun-1.14-1.module_el8+871+552be958.x86_64 5/13 2024-02-19T22:02:06.449 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 6/13 2024-02-19T22:02:06.449 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-02-19T22:02:06.449 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-02-19T22:02:06.449 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-02-19T22:02:06.450 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-02-19T22:02:06.450 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-19T22:02:06.450 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-19T22:02:07.123 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-02-19T22:02:07.124 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:02:07.124 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-02-19T22:02:07.124 INFO:teuthology.orchestra.run.smithi052.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-02-19T22:02:07.124 INFO:teuthology.orchestra.run.smithi052.stdout: buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2024-02-19T22:02:07.124 INFO:teuthology.orchestra.run.smithi052.stdout: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 2024-02-19T22:02:07.124 INFO:teuthology.orchestra.run.smithi052.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-02-19T22:02:07.124 INFO:teuthology.orchestra.run.smithi052.stdout: crun-1.14-1.module_el8+871+552be958.x86_64 2024-02-19T22:02:07.124 INFO:teuthology.orchestra.run.smithi052.stdout: netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 2024-02-19T22:02:07.124 INFO:teuthology.orchestra.run.smithi052.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-02-19T22:02:07.124 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-19T22:02:07.124 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-19T22:02:07.124 INFO:teuthology.orchestra.run.smithi052.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-02-19T22:02:07.124 INFO:teuthology.orchestra.run.smithi052.stdout: toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 2024-02-19T22:02:07.125 INFO:teuthology.orchestra.run.smithi052.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-02-19T22:02:07.125 INFO:teuthology.orchestra.run.smithi052.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-19T22:02:07.125 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:02:07.125 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-02-19T22:02:07.396 DEBUG:teuthology.parallel:result is None 2024-02-19T22:02:07.397 INFO:teuthology.run_tasks:Running task pexec... 2024-02-19T22:02:07.405 INFO:teuthology.task.pexec:Executing custom commands... 2024-02-19T22:02:07.405 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi052.front.sepia.ceph.com 2024-02-19T22:02:07.405 DEBUG:teuthology.orchestra.run.smithi052:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-19T22:02:07.406 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi088.front.sepia.ceph.com 2024-02-19T22:02:07.406 DEBUG:teuthology.orchestra.run.smithi088:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-19T22:02:07.589 DEBUG:teuthology.parallel:result is None 2024-02-19T22:02:07.590 DEBUG:teuthology.parallel:result is None 2024-02-19T22:02:07.590 INFO:teuthology.run_tasks:Running task install... 2024-02-19T22:02:07.598 DEBUG:teuthology.task.install:project ceph 2024-02-19T22:02:07.599 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '81bd20d634209c7cb82c18be12b4b5a05643ebf1'}} 2024-02-19T22:02:07.599 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'flavor': 'default', 'sha1': '81bd20d634209c7cb82c18be12b4b5a05643ebf1'} 2024-02-19T22:02:07.599 INFO:teuthology.task.install:Using flavor: default 2024-02-19T22:02:07.607 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-19T22:02:07.607 INFO:teuthology.task.install:extra packages: [] 2024-02-19T22:02:07.607 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '81bd20d634209c7cb82c18be12b4b5a05643ebf1', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-02-19T22:02:07.607 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T22:02:07.608 INFO:teuthology.packaging:ref: None 2024-02-19T22:02:07.608 INFO:teuthology.packaging:tag: v16.2.4 2024-02-19T22:02:07.608 INFO:teuthology.packaging:branch: None 2024-02-19T22:02:07.608 INFO:teuthology.packaging:sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T22:02:07.871 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-19T22:02:07.872 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-19T22:02:07.875 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '81bd20d634209c7cb82c18be12b4b5a05643ebf1', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-02-19T22:02:07.875 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T22:02:07.875 INFO:teuthology.packaging:ref: None 2024-02-19T22:02:07.875 INFO:teuthology.packaging:tag: v16.2.4 2024-02-19T22:02:07.875 INFO:teuthology.packaging:branch: None 2024-02-19T22:02:07.875 INFO:teuthology.packaging:sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T22:02:07.875 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-19T22:02:08.026 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-02-19T22:02:08.026 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-02-19T22:02:08.061 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-19T22:02:08.061 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-02-19T22:02:08.061 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-19T22:02:08.094 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-19T22:02:08.095 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T22:02:08.095 INFO:teuthology.packaging:ref: None 2024-02-19T22:02:08.095 INFO:teuthology.packaging:tag: v16.2.4 2024-02-19T22:02:08.095 INFO:teuthology.packaging:branch: None 2024-02-19T22:02:08.095 INFO:teuthology.packaging:sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T22:02:08.095 DEBUG:teuthology.orchestra.run.smithi088:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-19T22:02:08.114 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-02-19T22:02:08.114 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-02-19T22:02:08.139 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-19T22:02:08.139 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-02-19T22:02:08.139 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-19T22:02:08.166 DEBUG:teuthology.orchestra.run.smithi088:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-19T22:02:08.170 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-19T22:02:08.170 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T22:02:08.171 INFO:teuthology.packaging:ref: None 2024-02-19T22:02:08.171 INFO:teuthology.packaging:tag: v16.2.4 2024-02-19T22:02:08.171 INFO:teuthology.packaging:branch: None 2024-02-19T22:02:08.171 INFO:teuthology.packaging:sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T22:02:08.171 DEBUG:teuthology.orchestra.run.smithi052:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-19T22:02:08.242 DEBUG:teuthology.orchestra.run.smithi052:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-19T22:02:08.246 DEBUG:teuthology.orchestra.run.smithi088:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-19T22:02:08.315 INFO:teuthology.orchestra.run.smithi088.stdout:check_obsoletes = 1 2024-02-19T22:02:08.316 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum clean all 2024-02-19T22:02:08.322 DEBUG:teuthology.orchestra.run.smithi052:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-19T22:02:08.389 INFO:teuthology.orchestra.run.smithi052.stdout:check_obsoletes = 1 2024-02-19T22:02:08.391 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum clean all 2024-02-19T22:02:08.585 INFO:teuthology.orchestra.run.smithi088.stdout:76 files removed 2024-02-19T22:02:08.614 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-radosgw 2024-02-19T22:02:08.661 INFO:teuthology.orchestra.run.smithi052.stdout:76 files removed 2024-02-19T22:02:08.690 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-radosgw 2024-02-19T22:02:09.102 INFO:teuthology.orchestra.run.smithi088.stdout:ceph packages for x86_64 890 kB/s | 74 kB 00:00 2024-02-19T22:02:09.203 INFO:teuthology.orchestra.run.smithi052.stdout:ceph packages for x86_64 908 kB/s | 74 kB 00:00 2024-02-19T22:02:09.219 INFO:teuthology.orchestra.run.smithi088.stdout:ceph noarch packages 191 kB/s | 15 kB 00:00 2024-02-19T22:02:09.320 INFO:teuthology.orchestra.run.smithi088.stdout:ceph source packages 24 kB/s | 1.7 kB 00:00 2024-02-19T22:02:09.336 INFO:teuthology.orchestra.run.smithi052.stdout:ceph noarch packages 163 kB/s | 15 kB 00:00 2024-02-19T22:02:09.446 INFO:teuthology.orchestra.run.smithi052.stdout:ceph source packages 23 kB/s | 1.7 kB 00:00 2024-02-19T22:02:22.987 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 8 - AppStream 2.0 MB/s | 27 MB 00:13 2024-02-19T22:02:25.985 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 8 - AppStream 1.6 MB/s | 27 MB 00:16 2024-02-19T22:02:29.021 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 8 - BaseOS 13 MB/s | 10 MB 00:00 2024-02-19T22:02:30.863 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 8 - Extras 140 kB/s | 18 kB 00:00 2024-02-19T22:02:31.439 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 8 - Extras common packages 15 kB/s | 6.9 kB 00:00 2024-02-19T22:02:31.734 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 8 - BaseOS 11 MB/s | 10 MB 00:00 2024-02-19T22:02:32.968 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 8 - PowerTools 3.4 MB/s | 4.8 MB 00:01 2024-02-19T22:02:33.510 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 8 - Extras 93 kB/s | 18 kB 00:00 2024-02-19T22:02:33.944 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 8 - Extras common packages 21 kB/s | 6.9 kB 00:00 2024-02-19T22:02:34.542 INFO:teuthology.orchestra.run.smithi052.stdout:Copr repo for python3-asyncssh owned by ceph 6.2 kB/s | 3.5 kB 00:00 2024-02-19T22:02:35.237 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 8 - PowerTools 4.0 MB/s | 4.8 MB 00:01 2024-02-19T22:02:35.862 INFO:teuthology.orchestra.run.smithi052.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2024-02-19T22:02:36.396 INFO:teuthology.orchestra.run.smithi088.stdout:Copr repo for python3-asyncssh owned by ceph 15 kB/s | 3.5 kB 00:00 2024-02-19T22:02:39.818 INFO:teuthology.orchestra.run.smithi052.stdout:lab-extras 550 kB/s | 24 kB 00:00 2024-02-19T22:02:43.620 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-02-19T22:02:43.623 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:02:43.623 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-02-19T22:02:43.623 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:02:43.624 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-02-19T22:02:43.624 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-02-19T22:02:43.624 INFO:teuthology.orchestra.run.smithi052.stdout:Upgrading: 2024-02-19T22:02:43.624 INFO:teuthology.orchestra.run.smithi052.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-19T22:02:43.624 INFO:teuthology.orchestra.run.smithi052.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-02-19T22:02:43.624 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-02-19T22:02:43.624 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-02-19T22:02:43.624 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-02-19T22:02:43.624 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-02-19T22:02:43.624 INFO:teuthology.orchestra.run.smithi052.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-02-19T22:02:43.624 INFO:teuthology.orchestra.run.smithi052.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-02-19T22:02:43.624 INFO:teuthology.orchestra.run.smithi052.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-19T22:02:43.624 INFO:teuthology.orchestra.run.smithi052.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-02-19T22:02:43.625 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-02-19T22:02:43.625 INFO:teuthology.orchestra.run.smithi052.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-19T22:02:43.625 INFO:teuthology.orchestra.run.smithi052.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-02-19T22:02:43.625 INFO:teuthology.orchestra.run.smithi052.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-02-19T22:02:43.625 INFO:teuthology.orchestra.run.smithi052.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-02-19T22:02:43.625 INFO:teuthology.orchestra.run.smithi052.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-02-19T22:02:43.625 INFO:teuthology.orchestra.run.smithi052.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-02-19T22:02:43.625 INFO:teuthology.orchestra.run.smithi052.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-19T22:02:43.625 INFO:teuthology.orchestra.run.smithi052.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-02-19T22:02:43.625 INFO:teuthology.orchestra.run.smithi052.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-02-19T22:02:43.625 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-02-19T22:02:43.625 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-02-19T22:02:43.625 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-02-19T22:02:43.626 INFO:teuthology.orchestra.run.smithi052.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-02-19T22:02:43.626 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-02-19T22:02:43.626 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-02-19T22:02:43.626 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-02-19T22:02:43.626 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:02:43.626 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-02-19T22:02:43.626 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:02:43.626 INFO:teuthology.orchestra.run.smithi052.stdout:Install 25 Packages 2024-02-19T22:02:43.626 INFO:teuthology.orchestra.run.smithi052.stdout:Upgrade 2 Packages 2024-02-19T22:02:43.626 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:02:43.627 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 59 M 2024-02-19T22:02:43.627 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-02-19T22:02:44.237 INFO:teuthology.orchestra.run.smithi052.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 37 MB/s | 6.0 MB 00:00 2024-02-19T22:02:44.262 INFO:teuthology.orchestra.run.smithi052.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 999 kB/s | 25 kB 00:00 2024-02-19T22:02:44.304 INFO:teuthology.orchestra.run.smithi052.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 19 MB/s | 801 kB 00:00 2024-02-19T22:02:44.338 INFO:teuthology.orchestra.run.smithi052.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2024-02-19T22:02:44.463 INFO:teuthology.orchestra.run.smithi052.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 33 MB/s | 4.1 MB 00:00 2024-02-19T22:02:44.488 INFO:teuthology.orchestra.run.smithi052.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.8 MB/s | 44 kB 00:00 2024-02-19T22:02:44.513 INFO:teuthology.orchestra.run.smithi052.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2024-02-19T22:02:44.539 INFO:teuthology.orchestra.run.smithi052.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.4 MB/s | 213 kB 00:00 2024-02-19T22:02:44.572 INFO:teuthology.orchestra.run.smithi052.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2024-02-19T22:02:44.614 INFO:teuthology.orchestra.run.smithi052.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 8.6 MB/s | 368 kB 00:00 2024-02-19T22:02:44.640 INFO:teuthology.orchestra.run.smithi052.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2024-02-19T22:02:44.782 INFO:teuthology.orchestra.run.smithi052.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 18 MB/s | 13 MB 00:00 2024-02-19T22:02:45.082 INFO:teuthology.orchestra.run.smithi052.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 24 MB/s | 24 MB 00:01 2024-02-19T22:02:45.107 INFO:teuthology.orchestra.run.smithi052.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 311 kB/s | 145 kB 00:00 2024-02-19T22:02:45.199 INFO:teuthology.orchestra.run.smithi052.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 478 kB/s | 44 kB 00:00 2024-02-19T22:02:45.349 INFO:teuthology.orchestra.run.smithi052.stdout:(16/27): librdkafka-1.6.1-1.el8.x86_64.rpm 1.2 MB/s | 677 kB 00:00 2024-02-19T22:02:45.400 INFO:teuthology.orchestra.run.smithi052.stdout:(17/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 818 kB/s | 259 kB 00:00 2024-02-19T22:02:45.700 INFO:teuthology.orchestra.run.smithi052.stdout:(18/27): daxctl-libs-71.1-7.el8.x86_64.rpm 84 kB/s | 42 kB 00:00 2024-02-19T22:02:45.725 INFO:teuthology.orchestra.run.smithi052.stdout:(19/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 125 kB/s | 47 kB 00:00 2024-02-19T22:02:45.867 INFO:teuthology.orchestra.run.smithi052.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 171 kB/s | 80 kB 00:00 2024-02-19T22:02:46.143 INFO:teuthology.orchestra.run.smithi052.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 263 kB/s | 116 kB 00:00 2024-02-19T22:02:46.260 INFO:teuthology.orchestra.run.smithi052.stdout:(22/27): gperftools-libs-2.7-9.el8.x86_64.rpm 573 kB/s | 306 kB 00:00 2024-02-19T22:02:46.277 INFO:teuthology.orchestra.run.smithi052.stdout:(23/27): liboath-2.6.2-3.el8.x86_64.rpm 439 kB/s | 59 kB 00:00 2024-02-19T22:02:46.387 INFO:teuthology.orchestra.run.smithi052.stdout:(24/27): librados2-16.2.4-0.el8.x86_64.rpm 34 MB/s | 3.7 MB 00:00 2024-02-19T22:02:46.419 INFO:teuthology.orchestra.run.smithi052.stdout:(25/27): leveldb-1.22-1.el8.x86_64.rpm 328 kB/s | 181 kB 00:00 2024-02-19T22:02:46.444 INFO:teuthology.orchestra.run.smithi052.stdout:(26/27): libunwind-1.3.1-3.el8.x86_64.rpm 408 kB/s | 75 kB 00:00 2024-02-19T22:02:46.519 INFO:teuthology.orchestra.run.smithi052.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 31 MB/s | 4.0 MB 00:00 2024-02-19T22:02:46.521 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:02:46.521 INFO:teuthology.orchestra.run.smithi052.stdout:Total 20 MB/s | 59 MB 00:02 2024-02-19T22:02:46.588 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-02-19T22:02:46.641 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-02-19T22:02:46.641 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-02-19T22:02:47.034 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-02-19T22:02:47.037 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-02-19T22:02:48.235 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-02-19T22:02:48.482 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-02-19T22:02:48.723 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-02-19T22:02:48.927 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-19T22:02:49.363 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-19T22:02:49.544 INFO:teuthology.orchestra.run.smithi052.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-19T22:02:49.641 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-19T22:02:49.870 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-02-19T22:02:50.035 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-19T22:02:50.100 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-19T22:02:50.350 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-02-19T22:02:50.927 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-02-19T22:02:51.101 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-19T22:02:51.177 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-19T22:02:51.390 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-02-19T22:02:51.593 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-19T22:02:51.664 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-19T22:02:51.876 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-02-19T22:02:52.195 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-02-19T22:02:52.385 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-19T22:02:52.449 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-19T22:02:52.585 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-19T22:02:53.066 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-19T22:02:53.285 INFO:teuthology.orchestra.run.smithi052.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-19T22:02:53.390 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-19T22:02:53.589 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-02-19T22:02:53.815 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-02-19T22:02:53.951 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-19T22:02:54.050 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-19T22:02:54.231 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-02-19T22:02:54.441 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-02-19T22:02:54.649 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-02-19T22:02:54.877 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-02-19T22:02:57.708 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-19T22:02:58.001 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-19T22:02:58.771 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-19T22:02:59.056 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-19T22:02:59.167 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-19T22:02:59.327 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-19T22:03:19.323 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-19T22:03:19.323 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /sys 2024-02-19T22:03:19.323 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /proc 2024-02-19T22:03:19.323 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /dev 2024-02-19T22:03:19.323 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /run 2024-02-19T22:03:19.324 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /mnt 2024-02-19T22:03:19.324 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /var/tmp 2024-02-19T22:03:19.324 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /home 2024-02-19T22:03:19.324 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /tmp 2024-02-19T22:03:19.324 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /dev 2024-02-19T22:03:19.324 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:03:20.736 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-19T22:03:20.816 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-19T22:03:20.816 INFO:teuthology.orchestra.run.smithi052.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-19T22:03:21.050 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-19T22:03:21.050 INFO:teuthology.orchestra.run.smithi052.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-19T22:03:21.753 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-19T22:03:21.753 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-02-19T22:03:21.753 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-02-19T22:03:21.753 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-02-19T22:03:21.753 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-02-19T22:03:21.753 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-02-19T22:03:21.753 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-02-19T22:03:21.753 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-02-19T22:03:21.754 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-02-19T22:03:21.754 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-02-19T22:03:21.754 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-02-19T22:03:21.754 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-02-19T22:03:21.754 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-02-19T22:03:21.754 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-02-19T22:03:21.754 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-02-19T22:03:21.754 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-02-19T22:03:21.754 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-02-19T22:03:21.754 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-02-19T22:03:21.754 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-02-19T22:03:21.755 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-02-19T22:03:21.755 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-02-19T22:03:21.755 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-02-19T22:03:21.755 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-02-19T22:03:21.755 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-02-19T22:03:21.755 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-02-19T22:03:21.755 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-02-19T22:03:21.755 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-02-19T22:03:21.755 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-02-19T22:03:21.755 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-02-19T22:03:22.739 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-02-19T22:03:22.739 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:03:22.739 INFO:teuthology.orchestra.run.smithi052.stdout:Upgraded: 2024-02-19T22:03:22.740 INFO:teuthology.orchestra.run.smithi052.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-02-19T22:03:22.740 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-02-19T22:03:22.740 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-02-19T22:03:22.740 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-02-19T22:03:22.740 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-02-19T22:03:22.740 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-02-19T22:03:22.740 INFO:teuthology.orchestra.run.smithi052.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-02-19T22:03:22.740 INFO:teuthology.orchestra.run.smithi052.stdout: fmt-6.2.1-1.el8.x86_64 2024-02-19T22:03:22.740 INFO:teuthology.orchestra.run.smithi052.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-19T22:03:22.740 INFO:teuthology.orchestra.run.smithi052.stdout: leveldb-1.22-1.el8.x86_64 2024-02-19T22:03:22.740 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-02-19T22:03:22.740 INFO:teuthology.orchestra.run.smithi052.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-19T22:03:22.740 INFO:teuthology.orchestra.run.smithi052.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-02-19T22:03:22.741 INFO:teuthology.orchestra.run.smithi052.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-02-19T22:03:22.741 INFO:teuthology.orchestra.run.smithi052.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-02-19T22:03:22.741 INFO:teuthology.orchestra.run.smithi052.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-02-19T22:03:22.741 INFO:teuthology.orchestra.run.smithi052.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-02-19T22:03:22.741 INFO:teuthology.orchestra.run.smithi052.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-19T22:03:22.741 INFO:teuthology.orchestra.run.smithi052.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-19T22:03:22.741 INFO:teuthology.orchestra.run.smithi052.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-02-19T22:03:22.742 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-02-19T22:03:22.742 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-02-19T22:03:22.742 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-02-19T22:03:22.742 INFO:teuthology.orchestra.run.smithi052.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-19T22:03:22.742 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-02-19T22:03:22.742 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-02-19T22:03:22.742 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-02-19T22:03:22.742 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:03:22.742 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-02-19T22:03:23.072 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-test 2024-02-19T22:03:23.857 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:44 ago on Mon 19 Feb 2024 10:02:39 PM UTC. 2024-02-19T22:03:25.527 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-02-19T22:03:25.528 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:03:25.528 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2024-02-19T22:03:25.528 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:03:25.528 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-02-19T22:03:25.528 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-02-19T22:03:25.529 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-02-19T22:03:25.529 INFO:teuthology.orchestra.run.smithi052.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-02-19T22:03:25.529 INFO:teuthology.orchestra.run.smithi052.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-02-19T22:03:25.529 INFO:teuthology.orchestra.run.smithi052.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-02-19T22:03:25.529 INFO:teuthology.orchestra.run.smithi052.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-02-19T22:03:25.529 INFO:teuthology.orchestra.run.smithi052.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-02-19T22:03:25.529 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:03:25.529 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-02-19T22:03:25.529 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:03:25.529 INFO:teuthology.orchestra.run.smithi052.stdout:Install 6 Packages 2024-02-19T22:03:25.529 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:03:25.530 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 53 M 2024-02-19T22:03:25.530 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 229 M 2024-02-19T22:03:25.530 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-02-19T22:03:25.775 INFO:teuthology.orchestra.run.smithi052.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 3.3 MB/s | 169 kB 00:00 2024-02-19T22:03:25.925 INFO:teuthology.orchestra.run.smithi052.stdout:(2/6): oniguruma-6.8.2-3.el8.x86_64.rpm 1.2 MB/s | 188 kB 00:00 2024-02-19T22:03:25.958 INFO:teuthology.orchestra.run.smithi052.stdout:(3/6): jq-1.6-8.el8.x86_64.rpm 868 kB/s | 203 kB 00:00 2024-02-19T22:03:26.109 INFO:teuthology.orchestra.run.smithi052.stdout:(4/6): socat-1.7.4.1-1.el8.x86_64.rpm 1.7 MB/s | 323 kB 00:00 2024-02-19T22:03:26.134 INFO:teuthology.orchestra.run.smithi052.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 402 kB/s | 71 kB 00:00 2024-02-19T22:03:26.626 INFO:teuthology.orchestra.run.smithi052.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 58 MB/s | 52 MB 00:00 2024-02-19T22:03:26.628 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:03:26.629 INFO:teuthology.orchestra.run.smithi052.stdout:Total 48 MB/s | 53 MB 00:01 2024-02-19T22:03:26.673 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-02-19T22:03:26.696 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-02-19T22:03:26.696 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-02-19T22:03:27.028 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-02-19T22:03:27.031 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-02-19T22:03:27.552 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-02-19T22:03:27.769 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-02-19T22:03:27.985 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-02-19T22:03:28.164 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-19T22:03:28.248 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-19T22:03:28.454 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-02-19T22:03:28.622 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-19T22:03:33.779 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-19T22:03:34.062 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-19T22:03:34.361 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-19T22:03:34.361 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-02-19T22:03:34.362 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-02-19T22:03:34.362 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-02-19T22:03:34.362 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-02-19T22:03:34.362 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-02-19T22:03:34.898 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-02-19T22:03:34.898 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:03:34.898 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-02-19T22:03:34.898 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-02-19T22:03:34.898 INFO:teuthology.orchestra.run.smithi052.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-02-19T22:03:34.898 INFO:teuthology.orchestra.run.smithi052.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-02-19T22:03:34.898 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:03:34.898 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-02-19T22:03:35.183 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph 2024-02-19T22:03:35.972 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:56 ago on Mon 19 Feb 2024 10:02:39 PM UTC. 2024-02-19T22:03:37.625 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-02-19T22:03:37.628 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:03:37.629 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-02-19T22:03:37.629 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:03:37.629 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-02-19T22:03:37.629 INFO:teuthology.orchestra.run.smithi052.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-02-19T22:03:37.629 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-02-19T22:03:37.629 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-02-19T22:03:37.629 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-02-19T22:03:37.629 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-02-19T22:03:37.629 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-02-19T22:03:37.629 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-02-19T22:03:37.629 INFO:teuthology.orchestra.run.smithi052.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-19T22:03:37.629 INFO:teuthology.orchestra.run.smithi052.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-19T22:03:37.630 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-02-19T22:03:37.630 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-19T22:03:37.630 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-19T22:03:37.630 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-02-19T22:03:37.630 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-19T22:03:37.630 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-19T22:03:37.630 INFO:teuthology.orchestra.run.smithi052.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-19T22:03:37.630 INFO:teuthology.orchestra.run.smithi052.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-02-19T22:03:37.630 INFO:teuthology.orchestra.run.smithi052.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-02-19T22:03:37.630 INFO:teuthology.orchestra.run.smithi052.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-19T22:03:37.630 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-19T22:03:37.630 INFO:teuthology.orchestra.run.smithi052.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-19T22:03:37.630 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-02-19T22:03:37.630 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-02-19T22:03:37.631 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-02-19T22:03:37.631 INFO:teuthology.orchestra.run.smithi052.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-19T22:03:37.631 INFO:teuthology.orchestra.run.smithi052.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-19T22:03:37.631 INFO:teuthology.orchestra.run.smithi052.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-19T22:03:37.631 INFO:teuthology.orchestra.run.smithi052.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-19T22:03:37.631 INFO:teuthology.orchestra.run.smithi052.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-19T22:03:37.631 INFO:teuthology.orchestra.run.smithi052.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-19T22:03:37.631 INFO:teuthology.orchestra.run.smithi052.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-19T22:03:37.631 INFO:teuthology.orchestra.run.smithi052.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-02-19T22:03:37.631 INFO:teuthology.orchestra.run.smithi052.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-19T22:03:37.631 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:03:37.631 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-02-19T22:03:37.631 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:03:37.631 INFO:teuthology.orchestra.run.smithi052.stdout:Install 32 Packages 2024-02-19T22:03:37.631 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:03:37.632 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 31 M 2024-02-19T22:03:37.632 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 111 M 2024-02-19T22:03:37.632 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-02-19T22:03:38.270 INFO:teuthology.orchestra.run.smithi052.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 91 kB/s | 6.3 kB 00:00 2024-02-19T22:03:38.328 INFO:teuthology.orchestra.run.smithi052.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-02-19T22:03:38.403 INFO:teuthology.orchestra.run.smithi052.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 12 MB/s | 2.4 MB 00:00 2024-02-19T22:03:38.437 INFO:teuthology.orchestra.run.smithi052.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 6.5 MB/s | 220 kB 00:00 2024-02-19T22:03:38.571 INFO:teuthology.orchestra.run.smithi052.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 17 MB/s | 5.0 MB 00:00 2024-02-19T22:03:38.604 INFO:teuthology.orchestra.run.smithi052.stdout:(6/32): python3-mako-1.0.6-14.el8.noarch.rpm 939 kB/s | 157 kB 00:00 2024-02-19T22:03:38.646 INFO:teuthology.orchestra.run.smithi052.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 2.4 MB/s | 103 kB 00:00 2024-02-19T22:03:38.663 INFO:teuthology.orchestra.run.smithi052.stdout:(8/32): python3-markupsafe-0.23-19.el8.x86_64.r 423 kB/s | 39 kB 00:00 2024-02-19T22:03:38.722 INFO:teuthology.orchestra.run.smithi052.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 760 kB/s | 57 kB 00:00 2024-02-19T22:03:38.939 INFO:teuthology.orchestra.run.smithi052.stdout:(10/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 30 MB/s | 18 MB 00:00 2024-02-19T22:03:38.964 INFO:teuthology.orchestra.run.smithi052.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.5 MB/s | 457 kB 00:00 2024-02-19T22:03:39.214 INFO:teuthology.orchestra.run.smithi052.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 505 kB/s | 249 kB 00:00 2024-02-19T22:03:39.240 INFO:teuthology.orchestra.run.smithi052.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 396 kB/s | 109 kB 00:00 2024-02-19T22:03:39.331 INFO:teuthology.orchestra.run.smithi052.stdout:(14/32): python3-cryptography-3.2.1-7.el8.x86_6 1.6 MB/s | 641 kB 00:00 2024-02-19T22:03:39.748 INFO:teuthology.orchestra.run.smithi052.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 82 kB/s | 44 kB 00:00 2024-02-19T22:03:39.840 INFO:teuthology.orchestra.run.smithi052.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 308 kB/s | 185 kB 00:00 2024-02-19T22:03:39.866 INFO:teuthology.orchestra.run.smithi052.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 323 kB/s | 173 kB 00:00 2024-02-19T22:03:39.974 INFO:teuthology.orchestra.run.smithi052.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 83 kB/s | 11 kB 00:00 2024-02-19T22:03:40.000 INFO:teuthology.orchestra.run.smithi052.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 131 kB/s | 18 kB 00:00 2024-02-19T22:03:40.100 INFO:teuthology.orchestra.run.smithi052.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 390 kB/s | 49 kB 00:00 2024-02-19T22:03:40.135 INFO:teuthology.orchestra.run.smithi052.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 440 kB/s | 59 kB 00:00 2024-02-19T22:03:40.208 INFO:teuthology.orchestra.run.smithi052.stdout:(22/32): python3-cherrypy-18.4.0-1.el8.noarch.r 839 kB/s | 384 kB 00:00 2024-02-19T22:03:40.276 INFO:teuthology.orchestra.run.smithi052.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 110 kB/s | 16 kB 00:00 2024-02-19T22:03:40.334 INFO:teuthology.orchestra.run.smithi052.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 147 kB/s | 19 kB 00:00 2024-02-19T22:03:40.401 INFO:teuthology.orchestra.run.smithi052.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 191 kB/s | 24 kB 00:00 2024-02-19T22:03:40.427 INFO:teuthology.orchestra.run.smithi052.stdout:(26/32): python3-pecan-1.3.2-9.el8.noarch.rpm 866 kB/s | 283 kB 00:00 2024-02-19T22:03:40.494 INFO:teuthology.orchestra.run.smithi052.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 184 kB/s | 29 kB 00:00 2024-02-19T22:03:40.535 INFO:teuthology.orchestra.run.smithi052.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 205 kB/s | 27 kB 00:00 2024-02-19T22:03:40.669 INFO:teuthology.orchestra.run.smithi052.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.0 MB/s | 255 kB 00:00 2024-02-19T22:03:40.694 INFO:teuthology.orchestra.run.smithi052.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 538 kB/s | 85 kB 00:00 2024-02-19T22:03:40.720 INFO:teuthology.orchestra.run.smithi052.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.1 MB/s | 251 kB 00:00 2024-02-19T22:03:40.803 INFO:teuthology.orchestra.run.smithi052.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 172 kB/s | 23 kB 00:00 2024-02-19T22:03:40.804 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:03:40.804 INFO:teuthology.orchestra.run.smithi052.stdout:Total 9.8 MB/s | 31 MB 00:03 2024-02-19T22:03:40.887 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-02-19T22:03:40.930 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-02-19T22:03:40.930 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-02-19T22:03:41.264 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-02-19T22:03:41.268 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-02-19T22:03:42.454 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-02-19T22:03:42.688 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-02-19T22:03:42.914 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-02-19T22:03:43.152 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-02-19T22:03:43.368 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-02-19T22:03:43.566 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-02-19T22:03:43.765 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-02-19T22:03:43.915 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-02-19T22:03:44.110 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-02-19T22:03:44.371 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-02-19T22:03:44.608 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-02-19T22:03:44.846 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-02-19T22:03:45.121 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-02-19T22:03:45.539 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-02-19T22:03:45.833 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-02-19T22:03:46.036 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-02-19T22:03:46.314 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-02-19T22:03:46.531 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-19T22:03:46.856 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-02-19T22:03:47.086 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-02-19T22:03:47.291 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-02-19T22:03:47.476 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-02-19T22:03:47.770 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-02-19T22:03:48.016 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-02-19T22:03:48.265 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-02-19T22:03:48.648 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-02-19T22:03:48.976 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-02-19T22:03:49.396 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-02-19T22:03:49.624 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-19T22:03:51.263 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-19T22:03:51.490 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-19T22:03:51.950 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-19T22:03:52.182 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-19T22:03:52.417 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-19T22:03:52.603 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-19T22:03:52.621 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-19T22:03:52.803 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-19T22:03:53.225 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-19T22:03:53.225 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-02-19T22:03:53.225 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-02-19T22:03:53.225 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-02-19T22:03:53.225 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-02-19T22:03:53.225 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-02-19T22:03:53.225 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-02-19T22:03:53.225 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-02-19T22:03:53.225 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-02-19T22:03:53.225 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-02-19T22:03:53.225 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-02-19T22:03:53.225 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-02-19T22:03:53.226 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-02-19T22:03:53.226 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-02-19T22:03:53.226 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-02-19T22:03:53.226 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-02-19T22:03:53.226 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-02-19T22:03:53.226 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-19T22:03:53.226 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-02-19T22:03:53.226 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-02-19T22:03:53.226 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-02-19T22:03:53.226 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-02-19T22:03:53.226 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-02-19T22:03:53.226 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-02-19T22:03:53.226 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-02-19T22:03:53.226 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-02-19T22:03:53.227 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-02-19T22:03:53.227 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-02-19T22:03:53.227 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-02-19T22:03:53.228 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-02-19T22:03:53.228 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-02-19T22:03:53.228 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-02-19T22:03:54.682 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-02-19T22:03:54.682 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:03:54.682 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-02-19T22:03:54.682 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-02-19T22:03:54.683 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-02-19T22:03:54.683 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-02-19T22:03:54.683 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-02-19T22:03:54.683 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-02-19T22:03:54.683 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-02-19T22:03:54.683 INFO:teuthology.orchestra.run.smithi052.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-19T22:03:54.683 INFO:teuthology.orchestra.run.smithi052.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-19T22:03:54.683 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-19T22:03:54.683 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-19T22:03:54.683 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-19T22:03:54.683 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-02-19T22:03:54.683 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-19T22:03:54.683 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-19T22:03:54.683 INFO:teuthology.orchestra.run.smithi052.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-19T22:03:54.683 INFO:teuthology.orchestra.run.smithi052.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-19T22:03:54.684 INFO:teuthology.orchestra.run.smithi052.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-19T22:03:54.684 INFO:teuthology.orchestra.run.smithi052.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-19T22:03:54.684 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-19T22:03:54.684 INFO:teuthology.orchestra.run.smithi052.stdout: python3-portend-2.6-1.el8.noarch 2024-02-19T22:03:54.684 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-19T22:03:54.684 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-19T22:03:54.684 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-19T22:03:54.684 INFO:teuthology.orchestra.run.smithi052.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-19T22:03:54.684 INFO:teuthology.orchestra.run.smithi052.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-19T22:03:54.684 INFO:teuthology.orchestra.run.smithi052.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-19T22:03:54.684 INFO:teuthology.orchestra.run.smithi052.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-19T22:03:54.684 INFO:teuthology.orchestra.run.smithi052.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-19T22:03:54.684 INFO:teuthology.orchestra.run.smithi052.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-19T22:03:54.685 INFO:teuthology.orchestra.run.smithi052.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-19T22:03:54.685 INFO:teuthology.orchestra.run.smithi052.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-19T22:03:54.685 INFO:teuthology.orchestra.run.smithi052.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-19T22:03:54.685 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:03:54.685 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-02-19T22:03:54.988 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-base 2024-02-19T22:03:55.781 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:16 ago on Mon 19 Feb 2024 10:02:39 PM UTC. 2024-02-19T22:03:57.386 INFO:teuthology.orchestra.run.smithi052.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:03:57.436 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-02-19T22:03:57.438 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-02-19T22:03:57.438 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-02-19T22:03:57.555 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install cephadm 2024-02-19T22:03:58.341 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:19 ago on Mon 19 Feb 2024 10:02:39 PM UTC. 2024-02-19T22:04:00.013 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-02-19T22:04:00.014 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:04:00.014 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2024-02-19T22:04:00.014 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:04:00.014 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-02-19T22:04:00.014 INFO:teuthology.orchestra.run.smithi052.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-02-19T22:04:00.014 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:04:00.014 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-02-19T22:04:00.014 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:04:00.014 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-02-19T22:04:00.014 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:04:00.015 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 71 k 2024-02-19T22:04:00.015 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 290 k 2024-02-19T22:04:00.015 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-02-19T22:04:00.373 INFO:teuthology.orchestra.run.smithi052.stdout:cephadm-16.2.4-0.el8.noarch.rpm 199 kB/s | 71 kB 00:00 2024-02-19T22:04:00.374 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:04:00.374 INFO:teuthology.orchestra.run.smithi052.stdout:Total 198 kB/s | 71 kB 00:00 2024-02-19T22:04:00.375 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-02-19T22:04:00.383 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-02-19T22:04:00.383 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-02-19T22:04:00.385 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-02-19T22:04:00.388 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-02-19T22:04:00.968 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-02-19T22:04:01.514 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T22:04:01.719 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T22:04:01.966 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T22:04:02.374 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T22:04:02.374 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:04:02.374 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-02-19T22:04:02.374 INFO:teuthology.orchestra.run.smithi052.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-02-19T22:04:02.374 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:04:02.374 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-02-19T22:04:02.617 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-immutable-object-cache 2024-02-19T22:04:03.409 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:24 ago on Mon 19 Feb 2024 10:02:39 PM UTC. 2024-02-19T22:04:05.075 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-02-19T22:04:05.075 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:04:05.076 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-02-19T22:04:05.076 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:04:05.076 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-02-19T22:04:05.076 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-02-19T22:04:05.076 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:04:05.076 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-02-19T22:04:05.076 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:04:05.076 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-02-19T22:04:05.076 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:04:05.077 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 163 k 2024-02-19T22:04:05.077 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 587 k 2024-02-19T22:04:05.077 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-02-19T22:04:05.113 INFO:teuthology.orchestra.run.smithi052.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 4.5 MB/s | 163 kB 00:00 2024-02-19T22:04:05.113 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:04:05.113 INFO:teuthology.orchestra.run.smithi052.stdout:Total 4.3 MB/s | 163 kB 00:00 2024-02-19T22:04:05.114 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-02-19T22:04:05.126 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-02-19T22:04:05.126 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-02-19T22:04:05.177 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-02-19T22:04:05.180 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-02-19T22:04:05.294 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-02-19T22:04:05.618 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:04:06.052 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:04:07.000 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:04:07.003 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:04:07.003 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-02-19T22:04:07.003 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-02-19T22:04:07.003 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:04:07.003 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-02-19T22:04:07.610 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr 2024-02-19T22:04:08.405 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:29 ago on Mon 19 Feb 2024 10:02:39 PM UTC. 2024-02-19T22:04:10.017 INFO:teuthology.orchestra.run.smithi052.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:04:10.067 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-02-19T22:04:10.068 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-02-19T22:04:10.068 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-02-19T22:04:10.171 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr-dashboard 2024-02-19T22:04:10.952 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:31 ago on Mon 19 Feb 2024 10:02:39 PM UTC. 2024-02-19T22:04:12.604 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-02-19T22:04:12.605 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:04:12.605 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-02-19T22:04:12.605 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:04:12.605 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-02-19T22:04:12.605 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-02-19T22:04:12.605 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-02-19T22:04:12.605 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-02-19T22:04:12.605 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-02-19T22:04:12.605 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-02-19T22:04:12.605 INFO:teuthology.orchestra.run.smithi052.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-02-19T22:04:12.605 INFO:teuthology.orchestra.run.smithi052.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-02-19T22:04:12.606 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:04:12.606 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-02-19T22:04:12.606 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:04:12.606 INFO:teuthology.orchestra.run.smithi052.stdout:Install 6 Packages 2024-02-19T22:04:12.606 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:04:12.606 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 3.0 M 2024-02-19T22:04:12.606 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 15 M 2024-02-19T22:04:12.606 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-02-19T22:04:12.971 INFO:teuthology.orchestra.run.smithi052.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 195 kB/s | 24 kB 00:00 2024-02-19T22:04:13.066 INFO:teuthology.orchestra.run.smithi052.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 41 kB/s | 8.9 kB 00:00 2024-02-19T22:04:13.317 INFO:teuthology.orchestra.run.smithi052.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 5.7 MB/s | 2.7 MB 00:00 2024-02-19T22:04:13.417 INFO:teuthology.orchestra.run.smithi052.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 96 kB/s | 34 kB 00:00 2024-02-19T22:04:13.476 INFO:teuthology.orchestra.run.smithi052.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.2 MB/s | 196 kB 00:00 2024-02-19T22:04:13.501 INFO:teuthology.orchestra.run.smithi052.stdout:(6/6): python3-jwt-1.6.1-2.el8.noarch.rpm 80 kB/s | 43 kB 00:00 2024-02-19T22:04:13.503 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:04:13.503 INFO:teuthology.orchestra.run.smithi052.stdout:Total 3.3 MB/s | 3.0 MB 00:00 2024-02-19T22:04:13.515 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-02-19T22:04:13.527 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-02-19T22:04:13.527 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-02-19T22:04:13.703 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-02-19T22:04:13.707 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-02-19T22:04:14.047 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-02-19T22:04:14.302 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-02-19T22:04:14.596 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-02-19T22:04:14.835 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-02-19T22:04:14.949 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-02-19T22:04:16.028 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-02-19T22:04:16.310 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-19T22:04:16.612 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-19T22:04:16.612 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-02-19T22:04:16.612 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-02-19T22:04:16.612 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-02-19T22:04:16.612 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-02-19T22:04:16.612 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-02-19T22:04:17.023 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-02-19T22:04:17.023 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:04:17.024 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-02-19T22:04:17.024 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-02-19T22:04:17.024 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-02-19T22:04:17.024 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-02-19T22:04:17.024 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-02-19T22:04:17.024 INFO:teuthology.orchestra.run.smithi052.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-02-19T22:04:17.024 INFO:teuthology.orchestra.run.smithi052.stdout: python3-routes-2.4.1-12.el8.noarch 2024-02-19T22:04:17.024 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:04:17.024 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-02-19T22:04:17.198 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-19T22:04:17.990 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:38 ago on Mon 19 Feb 2024 10:02:39 PM UTC. 2024-02-19T22:04:19.661 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-02-19T22:04:19.662 INFO:teuthology.orchestra.run.smithi052.stdout:============================================================================================ 2024-02-19T22:04:19.662 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-02-19T22:04:19.662 INFO:teuthology.orchestra.run.smithi052.stdout:============================================================================================ 2024-02-19T22:04:19.662 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-02-19T22:04:19.662 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-02-19T22:04:19.662 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-02-19T22:04:19.662 INFO:teuthology.orchestra.run.smithi052.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-02-19T22:04:19.662 INFO:teuthology.orchestra.run.smithi052.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-02-19T22:04:19.663 INFO:teuthology.orchestra.run.smithi052.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-02-19T22:04:19.663 INFO:teuthology.orchestra.run.smithi052.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-02-19T22:04:19.663 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-02-19T22:04:19.663 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-02-19T22:04:19.663 INFO:teuthology.orchestra.run.smithi052.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-02-19T22:04:19.663 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:04:19.663 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-02-19T22:04:19.663 INFO:teuthology.orchestra.run.smithi052.stdout:============================================================================================ 2024-02-19T22:04:19.663 INFO:teuthology.orchestra.run.smithi052.stdout:Install 8 Packages 2024-02-19T22:04:19.663 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:04:19.664 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 37 M 2024-02-19T22:04:19.664 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 210 M 2024-02-19T22:04:19.664 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-02-19T22:04:20.223 INFO:teuthology.orchestra.run.smithi052.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 49 MB/s | 8.6 MB 00:00 2024-02-19T22:04:21.307 INFO:teuthology.orchestra.run.smithi052.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 3.9 MB/s | 4.9 MB 00:01 2024-02-19T22:04:21.415 INFO:teuthology.orchestra.run.smithi052.stdout:(3/8): openblas-0.3.15-4.el8.x86_64.rpm 3.5 MB/s | 4.7 MB 00:01 2024-02-19T22:04:21.440 INFO:teuthology.orchestra.run.smithi052.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.7 MB/s | 225 kB 00:00 2024-02-19T22:04:21.808 INFO:teuthology.orchestra.run.smithi052.stdout:(5/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.7 MB/s | 645 kB 00:00 2024-02-19T22:04:21.912 INFO:teuthology.orchestra.run.smithi052.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 1.6 MB/s | 172 kB 00:00 2024-02-19T22:04:22.625 INFO:teuthology.orchestra.run.smithi052.stdout:(7/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 1.6 MB/s | 3.7 MB 00:02 2024-02-19T22:04:27.676 INFO:teuthology.orchestra.run.smithi052.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 2.3 MB/s | 14 MB 00:06 2024-02-19T22:04:27.677 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:04:27.677 INFO:teuthology.orchestra.run.smithi052.stdout:Total 4.6 MB/s | 37 MB 00:08 2024-02-19T22:04:27.915 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-02-19T22:04:27.935 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-02-19T22:04:27.935 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-02-19T22:04:28.281 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-02-19T22:04:28.284 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-02-19T22:04:28.958 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-02-19T22:04:29.097 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-19T22:04:29.307 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-19T22:04:29.502 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-19T22:04:30.132 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-19T22:04:30.322 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-02-19T22:04:30.953 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-02-19T22:04:31.150 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-19T22:04:31.886 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-19T22:04:32.246 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-02-19T22:04:32.508 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-19T22:04:34.372 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-19T22:04:35.745 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-02-19T22:04:35.995 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-19T22:04:36.323 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-19T22:04:36.324 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-02-19T22:04:36.324 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-02-19T22:04:36.324 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-02-19T22:04:36.325 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-02-19T22:04:36.325 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-02-19T22:04:36.325 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-02-19T22:04:36.325 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-02-19T22:04:36.809 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-02-19T22:04:36.810 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:04:36.810 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-02-19T22:04:36.810 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-02-19T22:04:36.810 INFO:teuthology.orchestra.run.smithi052.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-02-19T22:04:36.810 INFO:teuthology.orchestra.run.smithi052.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-02-19T22:04:36.810 INFO:teuthology.orchestra.run.smithi052.stdout: openblas-0.3.15-4.el8.x86_64 2024-02-19T22:04:36.811 INFO:teuthology.orchestra.run.smithi052.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-02-19T22:04:36.811 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-02-19T22:04:36.811 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-02-19T22:04:36.811 INFO:teuthology.orchestra.run.smithi052.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-02-19T22:04:36.811 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:04:36.811 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-02-19T22:04:36.937 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr-rook 2024-02-19T22:04:37.734 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:58 ago on Mon 19 Feb 2024 10:02:39 PM UTC. 2024-02-19T22:04:39.393 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-02-19T22:04:39.394 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:04:39.394 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-02-19T22:04:39.394 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:04:39.394 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-02-19T22:04:39.394 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-02-19T22:04:39.395 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-02-19T22:04:39.395 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-02-19T22:04:39.395 INFO:teuthology.orchestra.run.smithi052.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-02-19T22:04:39.395 INFO:teuthology.orchestra.run.smithi052.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-02-19T22:04:39.395 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-02-19T22:04:39.395 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-02-19T22:04:39.395 INFO:teuthology.orchestra.run.smithi052.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-02-19T22:04:39.395 INFO:teuthology.orchestra.run.smithi052.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-02-19T22:04:39.395 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-02-19T22:04:39.395 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-02-19T22:04:39.395 INFO:teuthology.orchestra.run.smithi052.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-02-19T22:04:39.395 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-02-19T22:04:39.395 INFO:teuthology.orchestra.run.smithi052.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-02-19T22:04:39.396 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:04:39.396 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-02-19T22:04:39.396 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:04:39.396 INFO:teuthology.orchestra.run.smithi052.stdout:Install 13 Packages 2024-02-19T22:04:39.396 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:04:39.396 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 1.9 M 2024-02-19T22:04:39.396 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 24 M 2024-02-19T22:04:39.397 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-02-19T22:04:39.672 INFO:teuthology.orchestra.run.smithi052.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 560 kB/s | 27 kB 00:00 2024-02-19T22:04:39.756 INFO:teuthology.orchestra.run.smithi052.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 152 kB/s | 20 kB 00:00 2024-02-19T22:04:39.781 INFO:teuthology.orchestra.run.smithi052.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 169 kB/s | 27 kB 00:00 2024-02-19T22:04:39.857 INFO:teuthology.orchestra.run.smithi052.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 687 kB/s | 126 kB 00:00 2024-02-19T22:04:39.913 INFO:teuthology.orchestra.run.smithi052.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 699 kB/s | 110 kB 00:00 2024-02-19T22:04:40.040 INFO:teuthology.orchestra.run.smithi052.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 598 kB/s | 155 kB 00:00 2024-02-19T22:04:40.074 INFO:teuthology.orchestra.run.smithi052.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 196 kB/s | 43 kB 00:00 2024-02-19T22:04:40.208 INFO:teuthology.orchestra.run.smithi052.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 112 kB/s | 33 kB 00:00 2024-02-19T22:04:40.275 INFO:teuthology.orchestra.run.smithi052.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 70 kB/s | 16 kB 00:00 2024-02-19T22:04:40.408 INFO:teuthology.orchestra.run.smithi052.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 243 kB/s | 81 kB 00:00 2024-02-19T22:04:40.467 INFO:teuthology.orchestra.run.smithi052.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 335 kB/s | 64 kB 00:00 2024-02-19T22:04:40.525 INFO:teuthology.orchestra.run.smithi052.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 524 kB/s | 61 kB 00:00 2024-02-19T22:04:40.559 INFO:teuthology.orchestra.run.smithi052.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 3.3 MB/s | 1.2 MB 00:00 2024-02-19T22:04:40.561 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:04:40.561 INFO:teuthology.orchestra.run.smithi052.stdout:Total 1.6 MB/s | 1.9 MB 00:01 2024-02-19T22:04:40.621 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-02-19T22:04:40.639 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-02-19T22:04:40.639 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-02-19T22:04:40.734 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-02-19T22:04:40.737 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-02-19T22:04:41.311 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-02-19T22:04:41.559 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-02-19T22:04:41.750 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-02-19T22:04:42.144 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-02-19T22:04:42.395 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-02-19T22:04:42.596 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-02-19T22:04:42.881 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-02-19T22:04:43.193 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-02-19T22:04:43.488 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-02-19T22:04:44.579 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-02-19T22:04:45.186 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-02-19T22:04:45.423 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-02-19T22:04:45.604 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-02-19T22:04:45.795 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-19T22:04:46.125 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-19T22:04:46.125 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-02-19T22:04:46.125 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-02-19T22:04:46.125 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-02-19T22:04:46.125 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-02-19T22:04:46.126 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-02-19T22:04:46.126 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-02-19T22:04:46.126 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-02-19T22:04:46.126 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-02-19T22:04:46.126 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-02-19T22:04:46.126 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-02-19T22:04:46.126 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-02-19T22:04:46.126 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-02-19T22:04:46.758 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-02-19T22:04:46.758 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:04:46.758 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-02-19T22:04:46.758 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-02-19T22:04:46.758 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-02-19T22:04:46.758 INFO:teuthology.orchestra.run.smithi052.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-02-19T22:04:46.758 INFO:teuthology.orchestra.run.smithi052.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-02-19T22:04:46.758 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-02-19T22:04:46.758 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-02-19T22:04:46.758 INFO:teuthology.orchestra.run.smithi052.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-02-19T22:04:46.758 INFO:teuthology.orchestra.run.smithi052.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-02-19T22:04:46.759 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-02-19T22:04:46.759 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-02-19T22:04:46.759 INFO:teuthology.orchestra.run.smithi052.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-02-19T22:04:46.759 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rsa-4.9-2.el8.noarch 2024-02-19T22:04:46.759 INFO:teuthology.orchestra.run.smithi052.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-02-19T22:04:46.759 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:04:46.759 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-02-19T22:04:46.891 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr-cephadm 2024-02-19T22:04:47.677 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:02:08 ago on Mon 19 Feb 2024 10:02:39 PM UTC. 2024-02-19T22:04:49.365 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-02-19T22:04:49.366 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:04:49.366 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-02-19T22:04:49.366 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:04:49.366 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-02-19T22:04:49.366 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-02-19T22:04:49.367 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-02-19T22:04:49.367 INFO:teuthology.orchestra.run.smithi052.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-02-19T22:04:49.367 INFO:teuthology.orchestra.run.smithi052.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-02-19T22:04:49.367 INFO:teuthology.orchestra.run.smithi052.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-02-19T22:04:49.367 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-02-19T22:04:49.367 INFO:teuthology.orchestra.run.smithi052.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-02-19T22:04:49.367 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:04:49.367 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-02-19T22:04:49.367 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:04:49.367 INFO:teuthology.orchestra.run.smithi052.stdout:Install 6 Packages 2024-02-19T22:04:49.367 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:04:49.368 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 5.9 M 2024-02-19T22:04:49.368 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 24 M 2024-02-19T22:04:49.368 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-02-19T22:04:49.633 INFO:teuthology.orchestra.run.smithi052.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 2.7 MB/s | 89 kB 00:00 2024-02-19T22:04:49.976 INFO:teuthology.orchestra.run.smithi052.stdout:(2/6): python3-jinja2-2.10.1-4.el8.noarch.rpm 1.5 MB/s | 578 kB 00:00 2024-02-19T22:04:50.110 INFO:teuthology.orchestra.run.smithi052.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 44 kB/s | 21 kB 00:00 2024-02-19T22:04:50.293 INFO:teuthology.orchestra.run.smithi052.stdout:(4/6): python3-remoto-1.2.1-1.el8.noarch.rpm 267 kB/s | 49 kB 00:00 2024-02-19T22:04:50.844 INFO:teuthology.orchestra.run.smithi052.stdout:(5/6): python3-execnet-1.7.1-1.el8.noarch.rpm 410 kB/s | 355 kB 00:00 2024-02-19T22:04:51.577 INFO:teuthology.orchestra.run.smithi052.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 2.4 MB/s | 4.8 MB 00:01 2024-02-19T22:04:51.579 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:04:51.580 INFO:teuthology.orchestra.run.smithi052.stdout:Total 2.6 MB/s | 5.9 MB 00:02 2024-02-19T22:04:51.635 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-02-19T22:04:51.646 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-02-19T22:04:51.646 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-02-19T22:04:51.760 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-02-19T22:04:51.763 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-02-19T22:04:52.224 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-02-19T22:04:52.515 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-02-19T22:04:52.843 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-02-19T22:04:53.833 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-02-19T22:04:54.328 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-02-19T22:04:54.592 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 5/6 2024-02-19T22:04:54.852 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-19T22:04:55.185 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-19T22:04:55.185 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-02-19T22:04:55.185 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-02-19T22:04:55.185 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 3/6 2024-02-19T22:04:55.185 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-02-19T22:04:55.186 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-02-19T22:04:55.618 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-02-19T22:04:55.618 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:04:55.619 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-02-19T22:04:55.619 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-02-19T22:04:55.619 INFO:teuthology.orchestra.run.smithi052.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-02-19T22:04:55.619 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jinja2-2.10.1-4.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-02-19T22:04:55.619 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:04:55.619 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-02-19T22:04:55.749 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-fuse 2024-02-19T22:04:56.546 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:02:17 ago on Mon 19 Feb 2024 10:02:39 PM UTC. 2024-02-19T22:04:58.211 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-02-19T22:04:58.211 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:04:58.211 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2024-02-19T22:04:58.212 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:04:58.212 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-02-19T22:04:58.212 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-02-19T22:04:58.212 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-02-19T22:04:58.212 INFO:teuthology.orchestra.run.smithi052.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-02-19T22:04:58.212 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:04:58.212 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-02-19T22:04:58.212 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:04:58.212 INFO:teuthology.orchestra.run.smithi052.stdout:Install 2 Packages 2024-02-19T22:04:58.212 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:04:58.213 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 968 k 2024-02-19T22:04:58.213 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 3.2 M 2024-02-19T22:04:58.213 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-02-19T22:04:58.461 INFO:teuthology.orchestra.run.smithi052.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 16 MB/s | 885 kB 00:00 2024-02-19T22:04:58.653 INFO:teuthology.orchestra.run.smithi052.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 337 kB/s | 83 kB 00:00 2024-02-19T22:04:58.654 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:04:58.654 INFO:teuthology.orchestra.run.smithi052.stdout:Total 2.1 MB/s | 968 kB 00:00 2024-02-19T22:04:58.665 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-02-19T22:04:58.687 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-02-19T22:04:58.687 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-02-19T22:04:58.782 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-02-19T22:04:58.785 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-02-19T22:04:58.984 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-02-19T22:04:59.265 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-02-19T22:04:59.469 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-19T22:04:59.869 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-19T22:04:59.869 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-02-19T22:05:00.487 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-02-19T22:05:00.487 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:05:00.487 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-02-19T22:05:00.487 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-02-19T22:05:00.488 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:05:00.488 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-02-19T22:05:00.746 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install librados-devel 2024-02-19T22:05:01.542 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:02:22 ago on Mon 19 Feb 2024 10:02:39 PM UTC. 2024-02-19T22:05:03.202 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-02-19T22:05:03.203 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:05:03.203 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2024-02-19T22:05:03.203 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:05:03.203 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-02-19T22:05:03.203 INFO:teuthology.orchestra.run.smithi052.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-02-19T22:05:03.203 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:05:03.203 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-02-19T22:05:03.203 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:05:03.203 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-02-19T22:05:03.203 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:05:03.203 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 131 k 2024-02-19T22:05:03.203 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 439 k 2024-02-19T22:05:03.204 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-02-19T22:05:03.249 INFO:teuthology.orchestra.run.smithi052.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.9 MB/s | 131 kB 00:00 2024-02-19T22:05:03.249 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:05:03.249 INFO:teuthology.orchestra.run.smithi052.stdout:Total 2.8 MB/s | 131 kB 00:00 2024-02-19T22:05:03.250 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-02-19T22:05:03.255 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-02-19T22:05:03.255 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-02-19T22:05:03.309 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-02-19T22:05:03.312 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-02-19T22:05:03.434 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-02-19T22:05:03.676 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:05:03.971 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:05:04.357 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:05:04.357 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:05:04.357 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-02-19T22:05:04.357 INFO:teuthology.orchestra.run.smithi052.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-02-19T22:05:04.357 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:05:04.357 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-02-19T22:05:04.626 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install libcephfs2 2024-02-19T22:05:05.426 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:02:26 ago on Mon 19 Feb 2024 10:02:39 PM UTC. 2024-02-19T22:05:07.039 INFO:teuthology.orchestra.run.smithi052.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:05:07.090 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-02-19T22:05:07.091 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-02-19T22:05:07.091 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-02-19T22:05:07.225 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install libcephfs-devel 2024-02-19T22:05:08.024 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:02:29 ago on Mon 19 Feb 2024 10:02:39 PM UTC. 2024-02-19T22:05:09.690 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-02-19T22:05:09.690 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:05:09.690 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2024-02-19T22:05:09.690 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:05:09.690 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-02-19T22:05:09.690 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-02-19T22:05:09.691 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:05:09.691 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-02-19T22:05:09.691 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:05:09.691 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-02-19T22:05:09.691 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:05:09.691 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 24 k 2024-02-19T22:05:09.691 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 94 k 2024-02-19T22:05:09.692 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-02-19T22:05:09.737 INFO:teuthology.orchestra.run.smithi052.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 528 kB/s | 24 kB 00:00 2024-02-19T22:05:09.738 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:05:09.738 INFO:teuthology.orchestra.run.smithi052.stdout:Total 514 kB/s | 24 kB 00:00 2024-02-19T22:05:09.738 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-02-19T22:05:09.742 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-02-19T22:05:09.742 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-02-19T22:05:09.745 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-02-19T22:05:09.748 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-02-19T22:05:09.796 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-02-19T22:05:10.030 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:05:10.311 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:05:10.417 INFO:teuthology.orchestra.run.smithi088.stdout:Extra Packages for Enterprise Linux 107 kB/s | 16 MB 02:33 2024-02-19T22:05:10.677 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:05:10.677 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:05:10.677 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-02-19T22:05:10.677 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-02-19T22:05:10.677 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:05:10.677 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-02-19T22:05:10.809 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install librados2 2024-02-19T22:05:11.611 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:02:32 ago on Mon 19 Feb 2024 10:02:39 PM UTC. 2024-02-19T22:05:13.219 INFO:teuthology.orchestra.run.smithi052.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:05:13.269 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-02-19T22:05:13.270 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-02-19T22:05:13.270 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-02-19T22:05:13.409 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install librbd1 2024-02-19T22:05:14.123 INFO:teuthology.orchestra.run.smithi088.stdout:lab-extras 594 kB/s | 24 kB 00:00 2024-02-19T22:05:14.202 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:02:35 ago on Mon 19 Feb 2024 10:02:39 PM UTC. 2024-02-19T22:05:15.826 INFO:teuthology.orchestra.run.smithi052.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:05:15.876 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-02-19T22:05:15.877 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-02-19T22:05:15.878 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-02-19T22:05:16.000 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-rados 2024-02-19T22:05:16.790 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:02:37 ago on Mon 19 Feb 2024 10:02:39 PM UTC. 2024-02-19T22:05:17.645 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-02-19T22:05:17.648 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:05:17.648 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-02-19T22:05:17.648 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:05:17.648 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-02-19T22:05:17.648 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-02-19T22:05:17.648 INFO:teuthology.orchestra.run.smithi088.stdout:Upgrading: 2024-02-19T22:05:17.648 INFO:teuthology.orchestra.run.smithi088.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-19T22:05:17.648 INFO:teuthology.orchestra.run.smithi088.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-02-19T22:05:17.648 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-02-19T22:05:17.648 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-02-19T22:05:17.648 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-02-19T22:05:17.648 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-02-19T22:05:17.648 INFO:teuthology.orchestra.run.smithi088.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-02-19T22:05:17.649 INFO:teuthology.orchestra.run.smithi088.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-02-19T22:05:17.649 INFO:teuthology.orchestra.run.smithi088.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-19T22:05:17.649 INFO:teuthology.orchestra.run.smithi088.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-02-19T22:05:17.649 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-02-19T22:05:17.649 INFO:teuthology.orchestra.run.smithi088.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-19T22:05:17.649 INFO:teuthology.orchestra.run.smithi088.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-02-19T22:05:17.649 INFO:teuthology.orchestra.run.smithi088.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-02-19T22:05:17.649 INFO:teuthology.orchestra.run.smithi088.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-02-19T22:05:17.649 INFO:teuthology.orchestra.run.smithi088.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-02-19T22:05:17.649 INFO:teuthology.orchestra.run.smithi088.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-02-19T22:05:17.649 INFO:teuthology.orchestra.run.smithi088.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-19T22:05:17.649 INFO:teuthology.orchestra.run.smithi088.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-02-19T22:05:17.649 INFO:teuthology.orchestra.run.smithi088.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-02-19T22:05:17.649 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-02-19T22:05:17.649 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-02-19T22:05:17.650 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-02-19T22:05:17.650 INFO:teuthology.orchestra.run.smithi088.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-02-19T22:05:17.650 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-02-19T22:05:17.650 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-02-19T22:05:17.650 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-02-19T22:05:17.650 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:05:17.650 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-02-19T22:05:17.650 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:05:17.650 INFO:teuthology.orchestra.run.smithi088.stdout:Install 25 Packages 2024-02-19T22:05:17.650 INFO:teuthology.orchestra.run.smithi088.stdout:Upgrade 2 Packages 2024-02-19T22:05:17.650 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:05:17.651 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 59 M 2024-02-19T22:05:17.651 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-02-19T22:05:18.324 INFO:teuthology.orchestra.run.smithi088.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 38 MB/s | 6.0 MB 00:00 2024-02-19T22:05:18.350 INFO:teuthology.orchestra.run.smithi088.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 996 kB/s | 25 kB 00:00 2024-02-19T22:05:18.383 INFO:teuthology.orchestra.run.smithi088.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 24 MB/s | 801 kB 00:00 2024-02-19T22:05:18.417 INFO:teuthology.orchestra.run.smithi088.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2024-02-19T22:05:18.417 INFO:teuthology.orchestra.run.smithi052.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:05:18.467 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-02-19T22:05:18.468 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-02-19T22:05:18.468 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-02-19T22:05:18.509 INFO:teuthology.orchestra.run.smithi088.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 44 MB/s | 4.1 MB 00:00 2024-02-19T22:05:18.534 INFO:teuthology.orchestra.run.smithi088.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.8 MB/s | 44 kB 00:00 2024-02-19T22:05:18.576 INFO:teuthology.orchestra.run.smithi088.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 1.8 MB/s | 80 kB 00:00 2024-02-19T22:05:18.577 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-rgw 2024-02-19T22:05:18.609 INFO:teuthology.orchestra.run.smithi088.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 6.4 MB/s | 213 kB 00:00 2024-02-19T22:05:18.643 INFO:teuthology.orchestra.run.smithi088.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2024-02-19T22:05:18.676 INFO:teuthology.orchestra.run.smithi088.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 11 MB/s | 368 kB 00:00 2024-02-19T22:05:18.702 INFO:teuthology.orchestra.run.smithi088.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.4 MB/s | 114 kB 00:00 2024-02-19T22:05:18.836 INFO:teuthology.orchestra.run.smithi088.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 19 MB/s | 13 MB 00:00 2024-02-19T22:05:19.111 INFO:teuthology.orchestra.run.smithi088.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 25 MB/s | 24 MB 00:00 2024-02-19T22:05:19.128 INFO:teuthology.orchestra.run.smithi088.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 341 kB/s | 145 kB 00:00 2024-02-19T22:05:19.161 INFO:teuthology.orchestra.run.smithi088.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 1.3 MB/s | 44 kB 00:00 2024-02-19T22:05:19.220 INFO:teuthology.orchestra.run.smithi088.stdout:(16/27): librdkafka-1.6.1-1.el8.x86_64.rpm 1.7 MB/s | 677 kB 00:00 2024-02-19T22:05:19.271 INFO:teuthology.orchestra.run.smithi088.stdout:(17/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.6 MB/s | 259 kB 00:00 2024-02-19T22:05:19.368 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:02:40 ago on Mon 19 Feb 2024 10:02:39 PM UTC. 2024-02-19T22:05:19.371 INFO:teuthology.orchestra.run.smithi088.stdout:(18/27): daxctl-libs-71.1-7.el8.x86_64.rpm 202 kB/s | 42 kB 00:00 2024-02-19T22:05:19.438 INFO:teuthology.orchestra.run.smithi088.stdout:(19/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 216 kB/s | 47 kB 00:00 2024-02-19T22:05:19.480 INFO:teuthology.orchestra.run.smithi088.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 384 kB/s | 80 kB 00:00 2024-02-19T22:05:19.622 INFO:teuthology.orchestra.run.smithi088.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 464 kB/s | 116 kB 00:00 2024-02-19T22:05:19.647 INFO:teuthology.orchestra.run.smithi088.stdout:(22/27): leveldb-1.22-1.el8.x86_64.rpm 1.1 MB/s | 181 kB 00:00 2024-02-19T22:05:19.672 INFO:teuthology.orchestra.run.smithi088.stdout:(23/27): gperftools-libs-2.7-9.el8.x86_64.rpm 1.3 MB/s | 306 kB 00:00 2024-02-19T22:05:19.689 INFO:teuthology.orchestra.run.smithi088.stdout:(24/27): liboath-2.6.2-3.el8.x86_64.rpm 876 kB/s | 59 kB 00:00 2024-02-19T22:05:19.765 INFO:teuthology.orchestra.run.smithi088.stdout:(25/27): librados2-16.2.4-0.el8.x86_64.rpm 41 MB/s | 3.7 MB 00:00 2024-02-19T22:05:19.790 INFO:teuthology.orchestra.run.smithi088.stdout:(26/27): libunwind-1.3.1-3.el8.x86_64.rpm 527 kB/s | 75 kB 00:00 2024-02-19T22:05:19.848 INFO:teuthology.orchestra.run.smithi088.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 26 MB/s | 4.0 MB 00:00 2024-02-19T22:05:19.853 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:05:19.853 INFO:teuthology.orchestra.run.smithi088.stdout:Total 27 MB/s | 59 MB 00:02 2024-02-19T22:05:19.920 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-02-19T22:05:19.968 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-02-19T22:05:19.969 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-02-19T22:05:20.320 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-02-19T22:05:20.323 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-02-19T22:05:20.979 INFO:teuthology.orchestra.run.smithi052.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:05:21.029 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-02-19T22:05:21.030 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-02-19T22:05:21.031 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-02-19T22:05:21.170 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-cephfs 2024-02-19T22:05:21.428 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-02-19T22:05:21.620 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-02-19T22:05:21.808 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-02-19T22:05:21.953 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:02:42 ago on Mon 19 Feb 2024 10:02:39 PM UTC. 2024-02-19T22:05:21.992 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-19T22:05:22.413 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-19T22:05:22.591 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-19T22:05:22.689 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-19T22:05:22.930 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-02-19T22:05:23.058 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-19T22:05:23.111 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-19T22:05:23.318 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-02-19T22:05:23.587 INFO:teuthology.orchestra.run.smithi052.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:05:23.637 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-02-19T22:05:23.639 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-02-19T22:05:23.639 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-02-19T22:05:23.745 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-rbd 2024-02-19T22:05:23.876 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-02-19T22:05:24.033 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-19T22:05:24.114 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-19T22:05:24.321 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-02-19T22:05:24.524 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-19T22:05:24.538 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:02:45 ago on Mon 19 Feb 2024 10:02:39 PM UTC. 2024-02-19T22:05:24.591 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-19T22:05:24.755 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-02-19T22:05:24.908 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-02-19T22:05:25.074 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-19T22:05:25.151 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-19T22:05:25.332 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-19T22:05:25.764 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-19T22:05:25.966 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-19T22:05:26.044 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-19T22:05:26.166 INFO:teuthology.orchestra.run.smithi052.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:05:26.217 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-02-19T22:05:26.218 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-02-19T22:05:26.218 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-02-19T22:05:26.252 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-02-19T22:05:26.318 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install rbd-fuse 2024-02-19T22:05:26.476 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-02-19T22:05:26.640 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-19T22:05:26.714 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-19T22:05:26.949 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-02-19T22:05:27.106 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:02:48 ago on Mon 19 Feb 2024 10:02:39 PM UTC. 2024-02-19T22:05:27.133 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-02-19T22:05:27.289 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-02-19T22:05:27.516 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-02-19T22:05:28.756 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-02-19T22:05:28.757 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:05:28.757 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2024-02-19T22:05:28.757 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:05:28.757 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-02-19T22:05:28.757 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-02-19T22:05:28.757 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:05:28.757 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-02-19T22:05:28.757 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:05:28.757 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-02-19T22:05:28.757 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:05:28.757 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 89 k 2024-02-19T22:05:28.758 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 298 k 2024-02-19T22:05:28.758 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-02-19T22:05:28.802 INFO:teuthology.orchestra.run.smithi052.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 2.0 MB/s | 89 kB 00:00 2024-02-19T22:05:28.803 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:05:28.803 INFO:teuthology.orchestra.run.smithi052.stdout:Total 1.9 MB/s | 89 kB 00:00 2024-02-19T22:05:28.803 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-02-19T22:05:28.814 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-02-19T22:05:28.814 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-02-19T22:05:28.868 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-02-19T22:05:28.871 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-02-19T22:05:28.993 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-02-19T22:05:29.173 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:05:29.510 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:05:29.920 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:05:29.921 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:05:29.921 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-02-19T22:05:29.921 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-02-19T22:05:29.921 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:05:29.921 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-02-19T22:05:30.077 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-19T22:05:30.173 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install rbd-mirror 2024-02-19T22:05:30.315 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-19T22:05:30.964 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:02:51 ago on Mon 19 Feb 2024 10:02:39 PM UTC. 2024-02-19T22:05:31.020 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-19T22:05:31.304 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-19T22:05:31.409 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-19T22:05:31.556 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-19T22:05:32.644 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-02-19T22:05:32.645 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:05:32.645 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2024-02-19T22:05:32.645 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:05:32.645 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-02-19T22:05:32.645 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-19T22:05:32.645 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:05:32.645 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-02-19T22:05:32.645 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:05:32.645 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-02-19T22:05:32.646 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:05:32.646 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 3.7 M 2024-02-19T22:05:32.646 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 18 M 2024-02-19T22:05:32.646 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-02-19T22:05:32.755 INFO:teuthology.orchestra.run.smithi052.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 34 MB/s | 3.7 MB 00:00 2024-02-19T22:05:32.755 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:05:32.755 INFO:teuthology.orchestra.run.smithi052.stdout:Total 34 MB/s | 3.7 MB 00:00 2024-02-19T22:05:32.756 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-02-19T22:05:32.769 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-02-19T22:05:32.769 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-02-19T22:05:32.836 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-02-19T22:05:32.839 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-02-19T22:05:33.336 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-02-19T22:05:33.564 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:05:34.005 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:05:34.455 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:05:34.455 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:05:34.455 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-02-19T22:05:34.455 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-02-19T22:05:34.455 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:05:34.455 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-02-19T22:05:34.758 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install rbd-nbd 2024-02-19T22:05:35.553 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:02:56 ago on Mon 19 Feb 2024 10:02:39 PM UTC. 2024-02-19T22:05:37.233 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-02-19T22:05:37.234 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:05:37.234 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2024-02-19T22:05:37.234 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:05:37.234 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-02-19T22:05:37.234 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-02-19T22:05:37.234 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:05:37.234 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-02-19T22:05:37.234 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-02-19T22:05:37.234 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-02-19T22:05:37.234 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:05:37.235 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 173 k 2024-02-19T22:05:37.235 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 649 k 2024-02-19T22:05:37.235 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-02-19T22:05:37.283 INFO:teuthology.orchestra.run.smithi052.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.6 MB/s | 173 kB 00:00 2024-02-19T22:05:37.283 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:05:37.283 INFO:teuthology.orchestra.run.smithi052.stdout:Total 3.5 MB/s | 173 kB 00:00 2024-02-19T22:05:37.284 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-02-19T22:05:37.296 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-02-19T22:05:37.296 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-02-19T22:05:37.349 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-02-19T22:05:37.352 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-02-19T22:05:37.467 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-02-19T22:05:37.700 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:05:38.001 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:05:38.573 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:05:38.573 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:05:38.573 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-02-19T22:05:38.573 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-02-19T22:05:38.573 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-02-19T22:05:38.574 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-02-19T22:05:38.825 DEBUG:teuthology.parallel:result is None 2024-02-19T22:05:50.648 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-19T22:05:50.648 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /sys 2024-02-19T22:05:50.648 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /proc 2024-02-19T22:05:50.648 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /dev 2024-02-19T22:05:50.649 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /run 2024-02-19T22:05:50.649 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /mnt 2024-02-19T22:05:50.649 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /var/tmp 2024-02-19T22:05:50.649 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /home 2024-02-19T22:05:50.649 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /tmp 2024-02-19T22:05:50.649 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /dev 2024-02-19T22:05:50.649 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:05:51.983 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-19T22:05:52.063 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-19T22:05:52.064 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-19T22:05:52.320 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-19T22:05:52.321 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-19T22:05:53.028 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-19T22:05:53.029 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-02-19T22:05:53.029 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-02-19T22:05:53.029 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-02-19T22:05:53.029 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-02-19T22:05:53.029 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-02-19T22:05:53.029 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-02-19T22:05:53.029 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-02-19T22:05:53.029 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-02-19T22:05:53.029 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-02-19T22:05:53.029 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-02-19T22:05:53.030 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-02-19T22:05:53.030 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-02-19T22:05:53.030 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-02-19T22:05:53.030 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-02-19T22:05:53.030 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-02-19T22:05:53.030 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-02-19T22:05:53.030 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-02-19T22:05:53.030 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-02-19T22:05:53.030 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-02-19T22:05:53.030 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-02-19T22:05:53.030 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-02-19T22:05:53.031 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-02-19T22:05:53.031 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-02-19T22:05:53.031 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-02-19T22:05:53.031 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-02-19T22:05:53.031 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-02-19T22:05:53.031 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-02-19T22:05:53.031 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-02-19T22:05:53.990 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-02-19T22:05:53.990 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:05:53.990 INFO:teuthology.orchestra.run.smithi088.stdout:Upgraded: 2024-02-19T22:05:53.990 INFO:teuthology.orchestra.run.smithi088.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-02-19T22:05:53.990 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-02-19T22:05:53.991 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-02-19T22:05:53.991 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-02-19T22:05:53.991 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-02-19T22:05:53.991 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-02-19T22:05:53.991 INFO:teuthology.orchestra.run.smithi088.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-02-19T22:05:53.991 INFO:teuthology.orchestra.run.smithi088.stdout: fmt-6.2.1-1.el8.x86_64 2024-02-19T22:05:53.992 INFO:teuthology.orchestra.run.smithi088.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-19T22:05:53.992 INFO:teuthology.orchestra.run.smithi088.stdout: leveldb-1.22-1.el8.x86_64 2024-02-19T22:05:53.992 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-02-19T22:05:53.992 INFO:teuthology.orchestra.run.smithi088.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-19T22:05:53.992 INFO:teuthology.orchestra.run.smithi088.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-02-19T22:05:53.992 INFO:teuthology.orchestra.run.smithi088.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-02-19T22:05:53.993 INFO:teuthology.orchestra.run.smithi088.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-02-19T22:05:53.993 INFO:teuthology.orchestra.run.smithi088.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-02-19T22:05:53.993 INFO:teuthology.orchestra.run.smithi088.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-02-19T22:05:53.993 INFO:teuthology.orchestra.run.smithi088.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-19T22:05:53.993 INFO:teuthology.orchestra.run.smithi088.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-19T22:05:53.993 INFO:teuthology.orchestra.run.smithi088.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-02-19T22:05:53.994 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-02-19T22:05:53.994 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-02-19T22:05:53.994 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-02-19T22:05:53.994 INFO:teuthology.orchestra.run.smithi088.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-19T22:05:53.994 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-02-19T22:05:53.994 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-02-19T22:05:53.995 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-02-19T22:05:53.995 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:05:53.995 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-02-19T22:05:54.257 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-test 2024-02-19T22:05:55.014 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:00:41 ago on Mon 19 Feb 2024 10:05:14 PM UTC. 2024-02-19T22:05:56.576 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-02-19T22:05:56.577 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:05:56.577 INFO:teuthology.orchestra.run.smithi088.stdout: Package Architecture Version Repository Size 2024-02-19T22:05:56.578 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:05:56.578 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-02-19T22:05:56.578 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-02-19T22:05:56.578 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-02-19T22:05:56.578 INFO:teuthology.orchestra.run.smithi088.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-02-19T22:05:56.578 INFO:teuthology.orchestra.run.smithi088.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-02-19T22:05:56.578 INFO:teuthology.orchestra.run.smithi088.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-02-19T22:05:56.578 INFO:teuthology.orchestra.run.smithi088.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-02-19T22:05:56.578 INFO:teuthology.orchestra.run.smithi088.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-02-19T22:05:56.578 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:05:56.578 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-02-19T22:05:56.578 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:05:56.578 INFO:teuthology.orchestra.run.smithi088.stdout:Install 6 Packages 2024-02-19T22:05:56.578 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:05:56.579 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 53 M 2024-02-19T22:05:56.579 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 229 M 2024-02-19T22:05:56.579 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-02-19T22:05:56.704 INFO:teuthology.orchestra.run.smithi088.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 2.0 MB/s | 169 kB 00:00 2024-02-19T22:05:56.746 INFO:teuthology.orchestra.run.smithi088.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 1.6 MB/s | 203 kB 00:00 2024-02-19T22:05:56.788 INFO:teuthology.orchestra.run.smithi088.stdout:(3/6): oniguruma-6.8.2-3.el8.x86_64.rpm 2.2 MB/s | 188 kB 00:00 2024-02-19T22:05:56.813 INFO:teuthology.orchestra.run.smithi088.stdout:(4/6): socat-1.7.4.1-1.el8.x86_64.rpm 4.7 MB/s | 323 kB 00:00 2024-02-19T22:05:56.914 INFO:teuthology.orchestra.run.smithi088.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 564 kB/s | 71 kB 00:00 2024-02-19T22:05:57.464 INFO:teuthology.orchestra.run.smithi088.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 62 MB/s | 52 MB 00:00 2024-02-19T22:05:57.466 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:05:57.466 INFO:teuthology.orchestra.run.smithi088.stdout:Total 60 MB/s | 53 MB 00:00 2024-02-19T22:05:57.508 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-02-19T22:05:57.529 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-02-19T22:05:57.529 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-02-19T22:05:57.840 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-02-19T22:05:57.843 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-02-19T22:05:58.321 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-02-19T22:05:58.536 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-02-19T22:05:58.743 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-02-19T22:05:58.900 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-19T22:05:58.968 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-19T22:05:59.183 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-02-19T22:05:59.334 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-19T22:06:04.204 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-19T22:06:04.431 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-19T22:06:04.710 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-19T22:06:04.711 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-02-19T22:06:04.711 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-02-19T22:06:04.711 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-02-19T22:06:04.711 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-02-19T22:06:04.711 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-02-19T22:06:05.161 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-02-19T22:06:05.161 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:06:05.161 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-02-19T22:06:05.161 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-02-19T22:06:05.161 INFO:teuthology.orchestra.run.smithi088.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-02-19T22:06:05.161 INFO:teuthology.orchestra.run.smithi088.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-02-19T22:06:05.161 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:06:05.161 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-02-19T22:06:05.368 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph 2024-02-19T22:06:06.125 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:00:52 ago on Mon 19 Feb 2024 10:05:14 PM UTC. 2024-02-19T22:06:07.686 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-02-19T22:06:07.689 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:06:07.689 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-02-19T22:06:07.689 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:06:07.689 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-02-19T22:06:07.689 INFO:teuthology.orchestra.run.smithi088.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-02-19T22:06:07.689 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-02-19T22:06:07.689 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-02-19T22:06:07.689 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-02-19T22:06:07.689 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-02-19T22:06:07.689 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-02-19T22:06:07.690 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-02-19T22:06:07.690 INFO:teuthology.orchestra.run.smithi088.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-19T22:06:07.690 INFO:teuthology.orchestra.run.smithi088.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-19T22:06:07.690 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-02-19T22:06:07.690 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-19T22:06:07.690 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-19T22:06:07.690 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-02-19T22:06:07.690 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-19T22:06:07.690 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-19T22:06:07.690 INFO:teuthology.orchestra.run.smithi088.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-19T22:06:07.690 INFO:teuthology.orchestra.run.smithi088.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-02-19T22:06:07.690 INFO:teuthology.orchestra.run.smithi088.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-02-19T22:06:07.690 INFO:teuthology.orchestra.run.smithi088.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-19T22:06:07.690 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-19T22:06:07.691 INFO:teuthology.orchestra.run.smithi088.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-19T22:06:07.691 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-02-19T22:06:07.691 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-02-19T22:06:07.691 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-02-19T22:06:07.691 INFO:teuthology.orchestra.run.smithi088.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-19T22:06:07.691 INFO:teuthology.orchestra.run.smithi088.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-19T22:06:07.691 INFO:teuthology.orchestra.run.smithi088.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-19T22:06:07.691 INFO:teuthology.orchestra.run.smithi088.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-19T22:06:07.691 INFO:teuthology.orchestra.run.smithi088.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-19T22:06:07.691 INFO:teuthology.orchestra.run.smithi088.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-19T22:06:07.691 INFO:teuthology.orchestra.run.smithi088.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-19T22:06:07.691 INFO:teuthology.orchestra.run.smithi088.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-02-19T22:06:07.691 INFO:teuthology.orchestra.run.smithi088.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-19T22:06:07.691 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:06:07.691 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-02-19T22:06:07.692 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:06:07.692 INFO:teuthology.orchestra.run.smithi088.stdout:Install 32 Packages 2024-02-19T22:06:07.692 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:06:07.693 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 31 M 2024-02-19T22:06:07.693 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 111 M 2024-02-19T22:06:07.693 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-02-19T22:06:08.257 INFO:teuthology.orchestra.run.smithi088.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 139 kB/s | 6.3 kB 00:00 2024-02-19T22:06:08.307 INFO:teuthology.orchestra.run.smithi088.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-02-19T22:06:08.382 INFO:teuthology.orchestra.run.smithi088.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 14 MB/s | 2.4 MB 00:00 2024-02-19T22:06:08.416 INFO:teuthology.orchestra.run.smithi088.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 6.5 MB/s | 220 kB 00:00 2024-02-19T22:06:08.491 INFO:teuthology.orchestra.run.smithi088.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 21 MB/s | 5.0 MB 00:00 2024-02-19T22:06:08.733 INFO:teuthology.orchestra.run.smithi088.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 43 MB/s | 18 MB 00:00 2024-02-19T22:06:08.758 INFO:teuthology.orchestra.run.smithi088.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 145 kB/s | 39 kB 00:00 2024-02-19T22:06:08.784 INFO:teuthology.orchestra.run.smithi088.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 427 kB/s | 157 kB 00:00 2024-02-19T22:06:08.825 INFO:teuthology.orchestra.run.smithi088.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 856 kB/s | 57 kB 00:00 2024-02-19T22:06:08.859 INFO:teuthology.orchestra.run.smithi088.stdout:(10/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 823 kB/s | 103 kB 00:00 2024-02-19T22:06:09.126 INFO:teuthology.orchestra.run.smithi088.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 828 kB/s | 249 kB 00:00 2024-02-19T22:06:09.160 INFO:teuthology.orchestra.run.smithi088.stdout:(12/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.2 MB/s | 457 kB 00:00 2024-02-19T22:06:09.202 INFO:teuthology.orchestra.run.smithi088.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 1.4 MB/s | 109 kB 00:00 2024-02-19T22:06:09.235 INFO:teuthology.orchestra.run.smithi088.stdout:(14/32): python3-cryptography-3.2.1-7.el8.x86_6 1.7 MB/s | 641 kB 00:00 2024-02-19T22:06:09.361 INFO:teuthology.orchestra.run.smithi088.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 217 kB/s | 44 kB 00:00 2024-02-19T22:06:09.578 INFO:teuthology.orchestra.run.smithi088.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 493 kB/s | 185 kB 00:00 2024-02-19T22:06:09.869 INFO:teuthology.orchestra.run.smithi088.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 38 kB/s | 11 kB 00:00 2024-02-19T22:06:09.887 INFO:teuthology.orchestra.run.smithi088.stdout:(18/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 267 kB/s | 173 kB 00:00 2024-02-19T22:06:09.912 INFO:teuthology.orchestra.run.smithi088.stdout:(19/32): python3-cherrypy-18.4.0-1.el8.noarch.r 698 kB/s | 384 kB 00:00 2024-02-19T22:06:09.937 INFO:teuthology.orchestra.run.smithi088.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 262 kB/s | 18 kB 00:00 2024-02-19T22:06:09.962 INFO:teuthology.orchestra.run.smithi088.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 644 kB/s | 49 kB 00:00 2024-02-19T22:06:09.988 INFO:teuthology.orchestra.run.smithi088.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 777 kB/s | 59 kB 00:00 2024-02-19T22:06:10.030 INFO:teuthology.orchestra.run.smithi088.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 233 kB/s | 16 kB 00:00 2024-02-19T22:06:10.055 INFO:teuthology.orchestra.run.smithi088.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 279 kB/s | 19 kB 00:00 2024-02-19T22:06:10.080 INFO:teuthology.orchestra.run.smithi088.stdout:(25/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.9 MB/s | 283 kB 00:00 2024-02-19T22:06:10.106 INFO:teuthology.orchestra.run.smithi088.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 316 kB/s | 24 kB 00:00 2024-02-19T22:06:10.131 INFO:teuthology.orchestra.run.smithi088.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 387 kB/s | 29 kB 00:00 2024-02-19T22:06:10.156 INFO:teuthology.orchestra.run.smithi088.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 365 kB/s | 27 kB 00:00 2024-02-19T22:06:10.206 INFO:teuthology.orchestra.run.smithi088.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 3.3 MB/s | 251 kB 00:00 2024-02-19T22:06:10.232 INFO:teuthology.orchestra.run.smithi088.stdout:(30/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.0 MB/s | 255 kB 00:00 2024-02-19T22:06:10.257 INFO:teuthology.orchestra.run.smithi088.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 850 kB/s | 85 kB 00:00 2024-02-19T22:06:10.282 INFO:teuthology.orchestra.run.smithi088.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 305 kB/s | 23 kB 00:00 2024-02-19T22:06:10.286 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:06:10.286 INFO:teuthology.orchestra.run.smithi088.stdout:Total 12 MB/s | 31 MB 00:02 2024-02-19T22:06:10.367 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-02-19T22:06:10.406 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-02-19T22:06:10.406 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-02-19T22:06:10.708 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-02-19T22:06:10.711 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-02-19T22:06:11.818 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-02-19T22:06:12.020 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-02-19T22:06:12.199 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-02-19T22:06:12.481 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-02-19T22:06:12.678 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-02-19T22:06:12.835 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-02-19T22:06:12.998 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-02-19T22:06:13.150 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-02-19T22:06:13.328 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-02-19T22:06:13.518 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-02-19T22:06:13.731 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-02-19T22:06:13.928 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-02-19T22:06:14.178 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-02-19T22:06:14.539 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-02-19T22:06:14.791 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-02-19T22:06:14.980 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-02-19T22:06:15.214 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-02-19T22:06:15.372 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-19T22:06:15.638 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-02-19T22:06:15.860 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-02-19T22:06:16.049 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-02-19T22:06:16.226 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-02-19T22:06:16.471 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-02-19T22:06:16.708 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-02-19T22:06:16.961 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-02-19T22:06:17.277 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-02-19T22:06:17.612 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-02-19T22:06:18.014 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-02-19T22:06:18.261 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-19T22:06:19.787 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-19T22:06:20.027 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-19T22:06:20.459 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-19T22:06:20.701 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-19T22:06:20.923 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-19T22:06:21.113 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-19T22:06:21.130 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-19T22:06:21.262 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-19T22:06:21.654 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-19T22:06:21.654 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-02-19T22:06:21.655 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-02-19T22:06:21.655 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-02-19T22:06:21.655 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-02-19T22:06:21.655 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-02-19T22:06:21.655 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-02-19T22:06:21.655 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-02-19T22:06:21.655 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-02-19T22:06:21.655 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-02-19T22:06:21.655 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-02-19T22:06:21.655 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-02-19T22:06:21.655 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-02-19T22:06:21.655 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-02-19T22:06:21.655 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-02-19T22:06:21.656 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-02-19T22:06:21.656 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-02-19T22:06:21.656 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-19T22:06:21.656 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-02-19T22:06:21.656 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-02-19T22:06:21.656 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-02-19T22:06:21.656 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-02-19T22:06:21.656 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-02-19T22:06:21.656 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-02-19T22:06:21.656 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-02-19T22:06:21.656 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-02-19T22:06:21.656 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-02-19T22:06:21.656 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-02-19T22:06:21.656 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-02-19T22:06:21.656 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-02-19T22:06:21.657 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-02-19T22:06:21.657 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-02-19T22:06:22.479 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-02-19T22:06:22.479 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:06:22.480 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-02-19T22:06:22.480 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-02-19T22:06:22.480 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-02-19T22:06:22.480 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-02-19T22:06:22.480 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-02-19T22:06:22.480 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-02-19T22:06:22.480 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-02-19T22:06:22.480 INFO:teuthology.orchestra.run.smithi088.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-19T22:06:22.480 INFO:teuthology.orchestra.run.smithi088.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-19T22:06:22.480 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-19T22:06:22.480 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-19T22:06:22.480 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-19T22:06:22.480 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-02-19T22:06:22.480 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-19T22:06:22.481 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-19T22:06:22.481 INFO:teuthology.orchestra.run.smithi088.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-19T22:06:22.481 INFO:teuthology.orchestra.run.smithi088.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-19T22:06:22.481 INFO:teuthology.orchestra.run.smithi088.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-19T22:06:22.481 INFO:teuthology.orchestra.run.smithi088.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-19T22:06:22.481 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-19T22:06:22.481 INFO:teuthology.orchestra.run.smithi088.stdout: python3-portend-2.6-1.el8.noarch 2024-02-19T22:06:22.481 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-19T22:06:22.481 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-19T22:06:22.481 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-19T22:06:22.481 INFO:teuthology.orchestra.run.smithi088.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-19T22:06:22.481 INFO:teuthology.orchestra.run.smithi088.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-19T22:06:22.481 INFO:teuthology.orchestra.run.smithi088.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-19T22:06:22.481 INFO:teuthology.orchestra.run.smithi088.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-19T22:06:22.481 INFO:teuthology.orchestra.run.smithi088.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-19T22:06:22.482 INFO:teuthology.orchestra.run.smithi088.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-19T22:06:22.482 INFO:teuthology.orchestra.run.smithi088.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-19T22:06:22.482 INFO:teuthology.orchestra.run.smithi088.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-19T22:06:22.482 INFO:teuthology.orchestra.run.smithi088.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-19T22:06:22.482 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:06:22.482 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-02-19T22:06:22.770 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-base 2024-02-19T22:06:23.521 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:09 ago on Mon 19 Feb 2024 10:05:14 PM UTC. 2024-02-19T22:06:25.015 INFO:teuthology.orchestra.run.smithi088.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:06:25.063 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-02-19T22:06:25.064 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-02-19T22:06:25.065 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-02-19T22:06:25.155 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install cephadm 2024-02-19T22:06:25.903 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:11 ago on Mon 19 Feb 2024 10:05:14 PM UTC. 2024-02-19T22:06:27.462 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-02-19T22:06:27.462 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:06:27.462 INFO:teuthology.orchestra.run.smithi088.stdout: Package Architecture Version Repository Size 2024-02-19T22:06:27.462 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:06:27.462 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-02-19T22:06:27.463 INFO:teuthology.orchestra.run.smithi088.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-02-19T22:06:27.463 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:06:27.463 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-02-19T22:06:27.463 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:06:27.463 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-02-19T22:06:27.463 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:06:27.463 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 71 k 2024-02-19T22:06:27.464 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 290 k 2024-02-19T22:06:27.464 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-02-19T22:06:27.503 INFO:teuthology.orchestra.run.smithi088.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.8 MB/s | 71 kB 00:00 2024-02-19T22:06:27.503 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:06:27.503 INFO:teuthology.orchestra.run.smithi088.stdout:Total 1.7 MB/s | 71 kB 00:00 2024-02-19T22:06:27.504 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-02-19T22:06:27.510 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-02-19T22:06:27.510 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-02-19T22:06:27.512 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-02-19T22:06:27.515 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-02-19T22:06:27.556 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-02-19T22:06:28.030 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T22:06:28.221 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T22:06:28.442 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T22:06:28.802 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T22:06:28.802 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:06:28.802 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-02-19T22:06:28.802 INFO:teuthology.orchestra.run.smithi088.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-02-19T22:06:28.802 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:06:28.802 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-02-19T22:06:29.065 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-immutable-object-cache 2024-02-19T22:06:29.816 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:15 ago on Mon 19 Feb 2024 10:05:14 PM UTC. 2024-02-19T22:06:31.342 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-02-19T22:06:31.342 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:06:31.342 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-02-19T22:06:31.342 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:06:31.342 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-02-19T22:06:31.342 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-02-19T22:06:31.343 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:06:31.343 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-02-19T22:06:31.343 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:06:31.343 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-02-19T22:06:31.343 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:06:31.343 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 163 k 2024-02-19T22:06:31.343 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 587 k 2024-02-19T22:06:31.344 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-02-19T22:06:31.938 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 274 kB/s | 163 kB 00:00 2024-02-19T22:06:31.938 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:06:31.939 INFO:teuthology.orchestra.run.smithi088.stdout:Total 274 kB/s | 163 kB 00:00 2024-02-19T22:06:31.939 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-02-19T22:06:31.951 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-02-19T22:06:31.952 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-02-19T22:06:32.004 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-02-19T22:06:32.008 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-02-19T22:06:32.124 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-02-19T22:06:32.335 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:06:32.767 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:06:33.141 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:06:33.141 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:06:33.142 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-02-19T22:06:33.142 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-02-19T22:06:33.142 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:06:33.142 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-02-19T22:06:33.372 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-mgr 2024-02-19T22:06:34.131 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:20 ago on Mon 19 Feb 2024 10:05:14 PM UTC. 2024-02-19T22:06:35.635 INFO:teuthology.orchestra.run.smithi088.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:06:35.682 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-02-19T22:06:35.683 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-02-19T22:06:35.683 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-02-19T22:06:35.777 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-mgr-dashboard 2024-02-19T22:06:36.524 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:22 ago on Mon 19 Feb 2024 10:05:14 PM UTC. 2024-02-19T22:06:38.038 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-02-19T22:06:38.039 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:06:38.039 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-02-19T22:06:38.039 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:06:38.039 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-02-19T22:06:38.040 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-02-19T22:06:38.040 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-02-19T22:06:38.040 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-02-19T22:06:38.040 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-02-19T22:06:38.040 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-02-19T22:06:38.040 INFO:teuthology.orchestra.run.smithi088.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-02-19T22:06:38.040 INFO:teuthology.orchestra.run.smithi088.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-02-19T22:06:38.040 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:06:38.040 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-02-19T22:06:38.040 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:06:38.041 INFO:teuthology.orchestra.run.smithi088.stdout:Install 6 Packages 2024-02-19T22:06:38.041 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:06:38.041 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 3.0 M 2024-02-19T22:06:38.041 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 15 M 2024-02-19T22:06:38.041 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-02-19T22:06:38.793 INFO:teuthology.orchestra.run.smithi088.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 448 kB/s | 24 kB 00:00 2024-02-19T22:06:38.818 INFO:teuthology.orchestra.run.smithi088.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 113 kB/s | 8.9 kB 00:00 2024-02-19T22:06:38.885 INFO:teuthology.orchestra.run.smithi088.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 18 MB/s | 2.7 MB 00:00 2024-02-19T22:06:39.086 INFO:teuthology.orchestra.run.smithi088.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 146 kB/s | 43 kB 00:00 2024-02-19T22:06:39.136 INFO:teuthology.orchestra.run.smithi088.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 106 kB/s | 34 kB 00:00 2024-02-19T22:06:39.369 INFO:teuthology.orchestra.run.smithi088.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 406 kB/s | 196 kB 00:00 2024-02-19T22:06:39.371 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:06:39.371 INFO:teuthology.orchestra.run.smithi088.stdout:Total 2.2 MB/s | 3.0 MB 00:01 2024-02-19T22:06:39.383 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-02-19T22:06:39.393 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-02-19T22:06:39.394 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-02-19T22:06:39.567 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-02-19T22:06:39.570 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-02-19T22:06:39.831 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-02-19T22:06:40.081 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-02-19T22:06:40.326 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-02-19T22:06:40.518 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-02-19T22:06:40.647 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-02-19T22:06:41.658 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-02-19T22:06:41.930 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-19T22:06:42.216 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-19T22:06:42.217 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-02-19T22:06:42.217 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-02-19T22:06:42.217 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-02-19T22:06:42.217 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-02-19T22:06:42.217 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-02-19T22:06:42.597 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-02-19T22:06:42.598 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:06:42.598 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-02-19T22:06:42.598 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-02-19T22:06:42.598 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-02-19T22:06:42.598 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-02-19T22:06:42.598 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-02-19T22:06:42.598 INFO:teuthology.orchestra.run.smithi088.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-02-19T22:06:42.598 INFO:teuthology.orchestra.run.smithi088.stdout: python3-routes-2.4.1-12.el8.noarch 2024-02-19T22:06:42.598 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:06:42.598 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-02-19T22:06:42.715 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-19T22:06:43.476 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:29 ago on Mon 19 Feb 2024 10:05:14 PM UTC. 2024-02-19T22:06:45.014 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-02-19T22:06:45.015 INFO:teuthology.orchestra.run.smithi088.stdout:============================================================================================ 2024-02-19T22:06:45.016 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-02-19T22:06:45.016 INFO:teuthology.orchestra.run.smithi088.stdout:============================================================================================ 2024-02-19T22:06:45.016 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-02-19T22:06:45.016 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-02-19T22:06:45.016 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-02-19T22:06:45.016 INFO:teuthology.orchestra.run.smithi088.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-02-19T22:06:45.016 INFO:teuthology.orchestra.run.smithi088.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-02-19T22:06:45.016 INFO:teuthology.orchestra.run.smithi088.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-02-19T22:06:45.016 INFO:teuthology.orchestra.run.smithi088.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-02-19T22:06:45.016 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-02-19T22:06:45.017 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-02-19T22:06:45.017 INFO:teuthology.orchestra.run.smithi088.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-02-19T22:06:45.017 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:06:45.017 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-02-19T22:06:45.017 INFO:teuthology.orchestra.run.smithi088.stdout:============================================================================================ 2024-02-19T22:06:45.017 INFO:teuthology.orchestra.run.smithi088.stdout:Install 8 Packages 2024-02-19T22:06:45.017 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:06:45.018 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 37 M 2024-02-19T22:06:45.018 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 210 M 2024-02-19T22:06:45.018 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-02-19T22:06:45.391 INFO:teuthology.orchestra.run.smithi088.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 62 MB/s | 8.6 MB 00:00 2024-02-19T22:06:45.899 INFO:teuthology.orchestra.run.smithi088.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 7.3 MB/s | 3.7 MB 00:00 2024-02-19T22:06:46.149 INFO:teuthology.orchestra.run.smithi088.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 901 kB/s | 225 kB 00:00 2024-02-19T22:06:47.433 INFO:teuthology.orchestra.run.smithi088.stdout:(4/8): openblas-0.3.15-4.el8.x86_64.rpm 2.2 MB/s | 4.7 MB 00:02 2024-02-19T22:06:47.791 INFO:teuthology.orchestra.run.smithi088.stdout:(5/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.8 MB/s | 645 kB 00:00 2024-02-19T22:06:47.867 INFO:teuthology.orchestra.run.smithi088.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 2.3 MB/s | 172 kB 00:00 2024-02-19T22:06:48.017 INFO:teuthology.orchestra.run.smithi088.stdout:(7/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 1.8 MB/s | 4.9 MB 00:02 2024-02-19T22:06:49.677 INFO:teuthology.orchestra.run.smithi088.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 4.0 MB/s | 14 MB 00:03 2024-02-19T22:06:49.678 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:06:49.678 INFO:teuthology.orchestra.run.smithi088.stdout:Total 8.0 MB/s | 37 MB 00:04 2024-02-19T22:06:49.911 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-02-19T22:06:49.929 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-02-19T22:06:49.930 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-02-19T22:06:50.257 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-02-19T22:06:50.260 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-02-19T22:06:50.868 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-02-19T22:06:50.997 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-19T22:06:51.128 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-19T22:06:51.295 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-19T22:06:51.895 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-19T22:06:52.089 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-02-19T22:06:52.708 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-02-19T22:06:52.857 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-19T22:06:53.536 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-19T22:06:53.857 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-02-19T22:06:54.115 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-19T22:06:55.856 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-19T22:06:57.168 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-02-19T22:06:57.389 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-19T22:06:57.692 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-19T22:06:57.692 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-02-19T22:06:57.692 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-02-19T22:06:57.692 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-02-19T22:06:57.692 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-02-19T22:06:57.692 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-02-19T22:06:57.692 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-02-19T22:06:57.692 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-02-19T22:06:58.094 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-02-19T22:06:58.094 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:06:58.094 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-02-19T22:06:58.094 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-02-19T22:06:58.094 INFO:teuthology.orchestra.run.smithi088.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-02-19T22:06:58.094 INFO:teuthology.orchestra.run.smithi088.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-02-19T22:06:58.094 INFO:teuthology.orchestra.run.smithi088.stdout: openblas-0.3.15-4.el8.x86_64 2024-02-19T22:06:58.094 INFO:teuthology.orchestra.run.smithi088.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-02-19T22:06:58.094 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-02-19T22:06:58.094 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-02-19T22:06:58.094 INFO:teuthology.orchestra.run.smithi088.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-02-19T22:06:58.094 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:06:58.095 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-02-19T22:06:58.232 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-mgr-rook 2024-02-19T22:06:58.984 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:44 ago on Mon 19 Feb 2024 10:05:14 PM UTC. 2024-02-19T22:07:00.521 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-02-19T22:07:00.523 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:07:00.523 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-02-19T22:07:00.523 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:07:00.523 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-02-19T22:07:00.523 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-02-19T22:07:00.523 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-02-19T22:07:00.523 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-02-19T22:07:00.523 INFO:teuthology.orchestra.run.smithi088.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-02-19T22:07:00.523 INFO:teuthology.orchestra.run.smithi088.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-02-19T22:07:00.523 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-02-19T22:07:00.523 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-02-19T22:07:00.523 INFO:teuthology.orchestra.run.smithi088.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-02-19T22:07:00.524 INFO:teuthology.orchestra.run.smithi088.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-02-19T22:07:00.524 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-02-19T22:07:00.524 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-02-19T22:07:00.524 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-02-19T22:07:00.524 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-02-19T22:07:00.524 INFO:teuthology.orchestra.run.smithi088.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-02-19T22:07:00.524 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:07:00.524 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-02-19T22:07:00.524 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:07:00.524 INFO:teuthology.orchestra.run.smithi088.stdout:Install 13 Packages 2024-02-19T22:07:00.524 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:07:00.525 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 1.9 M 2024-02-19T22:07:00.525 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 24 M 2024-02-19T22:07:00.525 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-02-19T22:07:01.108 INFO:teuthology.orchestra.run.smithi088.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 597 kB/s | 27 kB 00:00 2024-02-19T22:07:01.174 INFO:teuthology.orchestra.run.smithi088.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 178 kB/s | 20 kB 00:00 2024-02-19T22:07:01.200 INFO:teuthology.orchestra.run.smithi088.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 194 kB/s | 27 kB 00:00 2024-02-19T22:07:01.242 INFO:teuthology.orchestra.run.smithi088.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 944 kB/s | 126 kB 00:00 2024-02-19T22:07:01.267 INFO:teuthology.orchestra.run.smithi088.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 1.2 MB/s | 110 kB 00:00 2024-02-19T22:07:01.609 INFO:teuthology.orchestra.run.smithi088.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 116 kB/s | 43 kB 00:00 2024-02-19T22:07:01.651 INFO:teuthology.orchestra.run.smithi088.stdout:(7/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 344 kB/s | 155 kB 00:00 2024-02-19T22:07:01.743 INFO:teuthology.orchestra.run.smithi088.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 69 kB/s | 33 kB 00:00 2024-02-19T22:07:01.976 INFO:teuthology.orchestra.run.smithi088.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 44 kB/s | 16 kB 00:00 2024-02-19T22:07:02.210 INFO:teuthology.orchestra.run.smithi088.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 145 kB/s | 81 kB 00:00 2024-02-19T22:07:02.302 INFO:teuthology.orchestra.run.smithi088.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 197 kB/s | 64 kB 00:00 2024-02-19T22:07:02.427 INFO:teuthology.orchestra.run.smithi088.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 283 kB/s | 61 kB 00:00 2024-02-19T22:07:02.636 INFO:teuthology.orchestra.run.smithi088.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 1.3 MB/s | 1.2 MB 00:00 2024-02-19T22:07:02.636 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:07:02.636 INFO:teuthology.orchestra.run.smithi088.stdout:Total 926 kB/s | 1.9 MB 00:02 2024-02-19T22:07:02.693 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-02-19T22:07:02.709 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-02-19T22:07:02.709 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-02-19T22:07:02.798 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-02-19T22:07:02.801 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-02-19T22:07:03.324 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-02-19T22:07:03.547 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-02-19T22:07:03.721 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-02-19T22:07:03.955 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-02-19T22:07:04.226 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-02-19T22:07:04.392 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-02-19T22:07:04.598 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-02-19T22:07:04.887 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-02-19T22:07:05.132 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-02-19T22:07:06.179 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-02-19T22:07:06.442 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-02-19T22:07:06.661 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-02-19T22:07:06.826 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-02-19T22:07:06.969 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-19T22:07:07.248 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-19T22:07:07.249 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-02-19T22:07:07.249 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-02-19T22:07:07.249 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-02-19T22:07:07.249 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-02-19T22:07:07.249 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-02-19T22:07:07.249 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-02-19T22:07:07.249 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-02-19T22:07:07.249 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-02-19T22:07:07.249 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-02-19T22:07:07.250 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-02-19T22:07:07.250 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-02-19T22:07:07.250 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-02-19T22:07:07.725 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-02-19T22:07:07.725 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:07:07.725 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-02-19T22:07:07.726 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-02-19T22:07:07.726 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-02-19T22:07:07.726 INFO:teuthology.orchestra.run.smithi088.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-02-19T22:07:07.726 INFO:teuthology.orchestra.run.smithi088.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-02-19T22:07:07.726 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-02-19T22:07:07.726 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-02-19T22:07:07.726 INFO:teuthology.orchestra.run.smithi088.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-02-19T22:07:07.726 INFO:teuthology.orchestra.run.smithi088.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-02-19T22:07:07.726 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-02-19T22:07:07.726 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-02-19T22:07:07.726 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-02-19T22:07:07.726 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rsa-4.9-2.el8.noarch 2024-02-19T22:07:07.726 INFO:teuthology.orchestra.run.smithi088.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-02-19T22:07:07.726 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:07:07.726 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-02-19T22:07:07.848 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-mgr-cephadm 2024-02-19T22:07:08.598 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:54 ago on Mon 19 Feb 2024 10:05:14 PM UTC. 2024-02-19T22:07:10.140 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-02-19T22:07:10.141 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:07:10.141 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-02-19T22:07:10.142 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:07:10.142 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-02-19T22:07:10.142 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-02-19T22:07:10.142 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-02-19T22:07:10.142 INFO:teuthology.orchestra.run.smithi088.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-02-19T22:07:10.142 INFO:teuthology.orchestra.run.smithi088.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-02-19T22:07:10.142 INFO:teuthology.orchestra.run.smithi088.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-02-19T22:07:10.142 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-02-19T22:07:10.142 INFO:teuthology.orchestra.run.smithi088.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-02-19T22:07:10.142 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:07:10.142 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-02-19T22:07:10.142 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:07:10.142 INFO:teuthology.orchestra.run.smithi088.stdout:Install 6 Packages 2024-02-19T22:07:10.142 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:07:10.143 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 5.9 M 2024-02-19T22:07:10.143 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 24 M 2024-02-19T22:07:10.143 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-02-19T22:07:10.261 INFO:teuthology.orchestra.run.smithi088.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.8 MB/s | 89 kB 00:00 2024-02-19T22:07:10.378 INFO:teuthology.orchestra.run.smithi088.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 182 kB/s | 21 kB 00:00 2024-02-19T22:07:10.546 INFO:teuthology.orchestra.run.smithi088.stdout:(3/6): python3-execnet-1.7.1-1.el8.noarch.rpm 2.1 MB/s | 355 kB 00:00 2024-02-19T22:07:10.604 INFO:teuthology.orchestra.run.smithi088.stdout:(4/6): python3-remoto-1.2.1-1.el8.noarch.rpm 839 kB/s | 49 kB 00:00 2024-02-19T22:07:10.754 INFO:teuthology.orchestra.run.smithi088.stdout:(5/6): python3-jinja2-2.10.1-4.el8.noarch.rpm 1.0 MB/s | 578 kB 00:00 2024-02-19T22:07:12.380 INFO:teuthology.orchestra.run.smithi088.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 2.2 MB/s | 4.8 MB 00:02 2024-02-19T22:07:12.380 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:07:12.381 INFO:teuthology.orchestra.run.smithi088.stdout:Total 2.6 MB/s | 5.9 MB 00:02 2024-02-19T22:07:12.434 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-02-19T22:07:12.444 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-02-19T22:07:12.445 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-02-19T22:07:12.545 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-02-19T22:07:12.548 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-02-19T22:07:12.824 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-02-19T22:07:13.091 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-02-19T22:07:13.401 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-02-19T22:07:14.320 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-02-19T22:07:14.648 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-02-19T22:07:14.896 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 5/6 2024-02-19T22:07:15.108 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-19T22:07:15.393 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-19T22:07:15.393 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-02-19T22:07:15.394 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-02-19T22:07:15.394 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 3/6 2024-02-19T22:07:15.394 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-02-19T22:07:15.394 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-02-19T22:07:16.085 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-02-19T22:07:16.085 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:07:16.085 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-02-19T22:07:16.085 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-02-19T22:07:16.085 INFO:teuthology.orchestra.run.smithi088.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-02-19T22:07:16.085 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jinja2-2.10.1-4.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-02-19T22:07:16.085 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:07:16.085 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-02-19T22:07:16.223 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-fuse 2024-02-19T22:07:16.976 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:02:02 ago on Mon 19 Feb 2024 10:05:14 PM UTC. 2024-02-19T22:07:18.513 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-02-19T22:07:18.513 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:07:18.513 INFO:teuthology.orchestra.run.smithi088.stdout: Package Architecture Version Repository Size 2024-02-19T22:07:18.513 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:07:18.513 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-02-19T22:07:18.514 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-02-19T22:07:18.514 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-02-19T22:07:18.514 INFO:teuthology.orchestra.run.smithi088.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-02-19T22:07:18.514 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:07:18.514 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-02-19T22:07:18.514 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:07:18.514 INFO:teuthology.orchestra.run.smithi088.stdout:Install 2 Packages 2024-02-19T22:07:18.514 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:07:18.515 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 968 k 2024-02-19T22:07:18.515 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 3.2 M 2024-02-19T22:07:18.515 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-02-19T22:07:18.649 INFO:teuthology.orchestra.run.smithi088.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 15 MB/s | 885 kB 00:00 2024-02-19T22:07:18.849 INFO:teuthology.orchestra.run.smithi088.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 324 kB/s | 83 kB 00:00 2024-02-19T22:07:18.850 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:07:18.850 INFO:teuthology.orchestra.run.smithi088.stdout:Total 2.8 MB/s | 968 kB 00:00 2024-02-19T22:07:18.859 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-02-19T22:07:18.880 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-02-19T22:07:18.880 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-02-19T22:07:18.965 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-02-19T22:07:18.968 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-02-19T22:07:19.146 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-02-19T22:07:19.407 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-02-19T22:07:19.582 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-19T22:07:19.973 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-19T22:07:19.973 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-02-19T22:07:20.349 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-02-19T22:07:20.349 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:07:20.349 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-02-19T22:07:20.349 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-02-19T22:07:20.349 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:07:20.349 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-02-19T22:07:20.591 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install librados-devel 2024-02-19T22:07:21.352 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:02:07 ago on Mon 19 Feb 2024 10:05:14 PM UTC. 2024-02-19T22:07:22.901 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-02-19T22:07:22.902 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:07:22.902 INFO:teuthology.orchestra.run.smithi088.stdout: Package Architecture Version Repository Size 2024-02-19T22:07:22.902 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:07:22.902 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-02-19T22:07:22.902 INFO:teuthology.orchestra.run.smithi088.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-02-19T22:07:22.902 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:07:22.903 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-02-19T22:07:22.903 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:07:22.903 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-02-19T22:07:22.903 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:07:22.903 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 131 k 2024-02-19T22:07:22.903 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 439 k 2024-02-19T22:07:22.903 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-02-19T22:07:22.952 INFO:teuthology.orchestra.run.smithi088.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.6 MB/s | 131 kB 00:00 2024-02-19T22:07:22.953 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:07:22.953 INFO:teuthology.orchestra.run.smithi088.stdout:Total 2.5 MB/s | 131 kB 00:00 2024-02-19T22:07:22.953 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-02-19T22:07:22.958 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-02-19T22:07:22.958 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-02-19T22:07:23.008 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-02-19T22:07:23.011 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-02-19T22:07:23.121 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-02-19T22:07:23.339 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:07:23.630 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:07:24.004 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:07:24.004 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:07:24.004 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-02-19T22:07:24.004 INFO:teuthology.orchestra.run.smithi088.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-02-19T22:07:24.004 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:07:24.005 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-02-19T22:07:24.231 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install libcephfs2 2024-02-19T22:07:25.007 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:02:11 ago on Mon 19 Feb 2024 10:05:14 PM UTC. 2024-02-19T22:07:26.485 INFO:teuthology.orchestra.run.smithi088.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:07:26.532 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-02-19T22:07:26.534 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-02-19T22:07:26.534 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-02-19T22:07:27.023 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install libcephfs-devel 2024-02-19T22:07:27.774 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:02:13 ago on Mon 19 Feb 2024 10:05:14 PM UTC. 2024-02-19T22:07:29.318 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-02-19T22:07:29.319 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:07:29.319 INFO:teuthology.orchestra.run.smithi088.stdout: Package Architecture Version Repository Size 2024-02-19T22:07:29.319 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:07:29.319 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-02-19T22:07:29.319 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-02-19T22:07:29.319 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:07:29.319 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-02-19T22:07:29.320 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:07:29.320 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-02-19T22:07:29.320 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:07:29.320 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 24 k 2024-02-19T22:07:29.320 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 94 k 2024-02-19T22:07:29.320 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-02-19T22:07:29.367 INFO:teuthology.orchestra.run.smithi088.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 526 kB/s | 24 kB 00:00 2024-02-19T22:07:29.367 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:07:29.367 INFO:teuthology.orchestra.run.smithi088.stdout:Total 511 kB/s | 24 kB 00:00 2024-02-19T22:07:29.368 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-02-19T22:07:29.371 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-02-19T22:07:29.371 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-02-19T22:07:29.373 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-02-19T22:07:29.376 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-02-19T22:07:29.429 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-02-19T22:07:29.650 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:07:29.912 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:07:30.254 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:07:30.254 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:07:30.254 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-02-19T22:07:30.254 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-02-19T22:07:30.255 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:07:30.255 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-02-19T22:07:30.372 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install librados2 2024-02-19T22:07:31.125 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:02:17 ago on Mon 19 Feb 2024 10:05:14 PM UTC. 2024-02-19T22:07:32.636 INFO:teuthology.orchestra.run.smithi088.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:07:32.682 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-02-19T22:07:32.683 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-02-19T22:07:32.683 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-02-19T22:07:32.780 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install librbd1 2024-02-19T22:07:33.526 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:02:19 ago on Mon 19 Feb 2024 10:05:14 PM UTC. 2024-02-19T22:07:35.028 INFO:teuthology.orchestra.run.smithi088.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:07:35.075 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-02-19T22:07:35.077 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-02-19T22:07:35.077 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-02-19T22:07:35.187 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install python3-rados 2024-02-19T22:07:35.945 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:02:21 ago on Mon 19 Feb 2024 10:05:14 PM UTC. 2024-02-19T22:07:37.433 INFO:teuthology.orchestra.run.smithi088.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:07:37.479 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-02-19T22:07:37.480 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-02-19T22:07:37.480 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-02-19T22:07:37.588 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install python3-rgw 2024-02-19T22:07:38.343 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:02:24 ago on Mon 19 Feb 2024 10:05:14 PM UTC. 2024-02-19T22:07:39.832 INFO:teuthology.orchestra.run.smithi088.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:07:39.878 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-02-19T22:07:39.880 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-02-19T22:07:39.880 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-02-19T22:07:39.981 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install python3-cephfs 2024-02-19T22:07:40.734 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:02:26 ago on Mon 19 Feb 2024 10:05:14 PM UTC. 2024-02-19T22:07:42.234 INFO:teuthology.orchestra.run.smithi088.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:07:42.281 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-02-19T22:07:42.282 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-02-19T22:07:42.282 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-02-19T22:07:42.384 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install python3-rbd 2024-02-19T22:07:43.141 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:02:29 ago on Mon 19 Feb 2024 10:05:14 PM UTC. 2024-02-19T22:07:44.655 INFO:teuthology.orchestra.run.smithi088.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:07:44.701 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-02-19T22:07:44.702 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-02-19T22:07:44.702 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-02-19T22:07:44.808 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install rbd-fuse 2024-02-19T22:07:45.555 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:02:31 ago on Mon 19 Feb 2024 10:05:14 PM UTC. 2024-02-19T22:07:47.106 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-02-19T22:07:47.106 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:07:47.106 INFO:teuthology.orchestra.run.smithi088.stdout: Package Architecture Version Repository Size 2024-02-19T22:07:47.106 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:07:47.106 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-02-19T22:07:47.106 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-02-19T22:07:47.106 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:07:47.107 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-02-19T22:07:47.107 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:07:47.107 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-02-19T22:07:47.107 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:07:47.107 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 89 k 2024-02-19T22:07:47.107 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 298 k 2024-02-19T22:07:47.107 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-02-19T22:07:47.154 INFO:teuthology.orchestra.run.smithi088.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.9 MB/s | 89 kB 00:00 2024-02-19T22:07:47.154 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:07:47.154 INFO:teuthology.orchestra.run.smithi088.stdout:Total 1.8 MB/s | 89 kB 00:00 2024-02-19T22:07:47.155 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-02-19T22:07:47.164 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-02-19T22:07:47.164 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-02-19T22:07:47.214 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-02-19T22:07:47.217 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-02-19T22:07:47.319 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-02-19T22:07:47.517 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:07:47.792 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:07:48.147 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:07:48.148 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:07:48.148 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-02-19T22:07:48.148 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-02-19T22:07:48.148 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:07:48.148 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-02-19T22:07:48.374 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install rbd-mirror 2024-02-19T22:07:49.143 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:02:35 ago on Mon 19 Feb 2024 10:05:14 PM UTC. 2024-02-19T22:07:50.694 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-02-19T22:07:50.695 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:07:50.695 INFO:teuthology.orchestra.run.smithi088.stdout: Package Architecture Version Repository Size 2024-02-19T22:07:50.695 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:07:50.695 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-02-19T22:07:50.695 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-19T22:07:50.695 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:07:50.695 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-02-19T22:07:50.695 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:07:50.695 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-02-19T22:07:50.695 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:07:50.696 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 3.7 M 2024-02-19T22:07:50.696 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 18 M 2024-02-19T22:07:50.696 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-02-19T22:07:50.791 INFO:teuthology.orchestra.run.smithi088.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 39 MB/s | 3.7 MB 00:00 2024-02-19T22:07:50.791 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:07:50.792 INFO:teuthology.orchestra.run.smithi088.stdout:Total 39 MB/s | 3.7 MB 00:00 2024-02-19T22:07:50.792 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-02-19T22:07:50.804 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-02-19T22:07:50.804 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-02-19T22:07:50.866 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-02-19T22:07:50.869 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-02-19T22:07:51.328 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-02-19T22:07:51.567 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:07:51.986 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:07:52.408 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:07:52.409 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:07:52.409 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-02-19T22:07:52.409 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-02-19T22:07:52.409 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:07:52.409 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-02-19T22:07:52.629 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install rbd-nbd 2024-02-19T22:07:53.391 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:02:39 ago on Mon 19 Feb 2024 10:05:14 PM UTC. 2024-02-19T22:07:54.954 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-02-19T22:07:54.955 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:07:54.955 INFO:teuthology.orchestra.run.smithi088.stdout: Package Architecture Version Repository Size 2024-02-19T22:07:54.955 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:07:54.955 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-02-19T22:07:54.955 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-02-19T22:07:54.955 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:07:54.955 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-02-19T22:07:54.956 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-02-19T22:07:54.956 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-02-19T22:07:54.956 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:07:54.956 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 173 k 2024-02-19T22:07:54.956 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 649 k 2024-02-19T22:07:54.956 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-02-19T22:07:54.994 INFO:teuthology.orchestra.run.smithi088.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 4.4 MB/s | 173 kB 00:00 2024-02-19T22:07:54.994 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:07:54.995 INFO:teuthology.orchestra.run.smithi088.stdout:Total 4.3 MB/s | 173 kB 00:00 2024-02-19T22:07:54.995 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-02-19T22:07:55.005 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-02-19T22:07:55.005 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-02-19T22:07:55.056 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-02-19T22:07:55.059 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-02-19T22:07:55.170 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-02-19T22:07:55.385 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:07:55.675 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:07:56.040 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:07:56.040 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:07:56.040 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-02-19T22:07:56.040 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-02-19T22:07:56.040 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:07:56.040 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-02-19T22:07:56.245 DEBUG:teuthology.parallel:result is None 2024-02-19T22:07:56.246 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T22:07:56.246 INFO:teuthology.packaging:ref: None 2024-02-19T22:07:56.246 INFO:teuthology.packaging:tag: v16.2.4 2024-02-19T22:07:56.246 INFO:teuthology.packaging:branch: None 2024-02-19T22:07:56.246 INFO:teuthology.packaging:sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T22:07:56.246 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-19T22:07:56.398 DEBUG:teuthology.orchestra.run.smithi052:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-19T22:07:56.426 INFO:teuthology.orchestra.run.smithi052.stdout:16.2.4-0.el8 2024-02-19T22:07:56.426 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-02-19T22:07:56.426 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-02-19T22:07:56.428 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T22:07:56.428 INFO:teuthology.packaging:ref: None 2024-02-19T22:07:56.428 INFO:teuthology.packaging:tag: v16.2.4 2024-02-19T22:07:56.428 INFO:teuthology.packaging:branch: None 2024-02-19T22:07:56.428 INFO:teuthology.packaging:sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T22:07:56.429 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-19T22:07:56.596 DEBUG:teuthology.orchestra.run.smithi088:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-19T22:07:56.619 INFO:teuthology.orchestra.run.smithi088.stdout:16.2.4-0.el8 2024-02-19T22:07:56.619 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-02-19T22:07:56.619 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-02-19T22:07:56.621 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-19T22:07:56.621 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-02-19T22:07:56.621 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-19T22:07:56.657 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-02-19T22:07:56.657 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-19T22:07:56.686 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-19T22:07:56.686 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-02-19T22:07:56.687 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/usr/bin/daemon-helper 2024-02-19T22:07:56.729 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-19T22:07:56.799 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-02-19T22:07:56.799 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/daemon-helper 2024-02-19T22:07:56.826 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-19T22:07:56.892 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-19T22:07:56.892 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-02-19T22:07:56.893 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-19T22:07:56.923 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-19T22:07:56.993 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-02-19T22:07:56.993 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-19T22:07:57.019 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-19T22:07:57.086 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-19T22:07:57.086 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-02-19T22:07:57.086 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/usr/bin/stdin-killer 2024-02-19T22:07:57.115 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-19T22:07:57.183 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-02-19T22:07:57.183 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/stdin-killer 2024-02-19T22:07:57.210 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-19T22:07:57.277 INFO:teuthology.run_tasks:Running task print... 2024-02-19T22:07:57.289 INFO:teuthology.task.print:**** done install task... 2024-02-19T22:07:57.289 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-19T22:07:57.366 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(PG_DEGRADED\\)', 'Degraded data redundancy', '\\(PG_', 'MDS_INSUFFICIENT_STANDBY', 'deprecated feature inline_data', 'compat changed unexpectedly', 'overall HEALTH_', 'OSD_DOWN', 'OSD_', 'but it is still running', 'is not responding'], 'sha1': '81bd20d634209c7cb82c18be12b4b5a05643ebf1'} 2024-02-19T22:07:57.366 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2024-02-19T22:07:57.367 INFO:tasks.cephadm:Cluster fsid is 56739a54-cf73-11ee-95bb-87774f69a715 2024-02-19T22:07:57.367 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-19T22:07:57.367 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-02-19T22:07:57.367 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi052': '172.21.15.52', 'mon.smithi088': '172.21.15.88'} 2024-02-19T22:07:57.367 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-19T22:07:57.367 DEBUG:teuthology.orchestra.run.smithi052:> sudo hostname $(hostname -s) 2024-02-19T22:07:57.397 DEBUG:teuthology.orchestra.run.smithi088:> sudo hostname $(hostname -s) 2024-02-19T22:07:57.424 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2024-02-19T22:07:57.424 DEBUG:teuthology.orchestra.run.smithi052:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-19T22:07:58.679 INFO:teuthology.orchestra.run.smithi052.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Feb 19 22:07 /home/ubuntu/cephtest/cephadm 2024-02-19T22:07:58.680 DEBUG:teuthology.orchestra.run.smithi088:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-19T22:07:59.050 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Feb 19 22:07 /home/ubuntu/cephtest/cephadm 2024-02-19T22:07:59.050 DEBUG:teuthology.orchestra.run.smithi052:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-19T22:07:59.074 DEBUG:teuthology.orchestra.run.smithi088:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-19T22:07:59.108 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /etc/ceph 2024-02-19T22:07:59.146 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /etc/ceph 2024-02-19T22:07:59.178 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 777 /etc/ceph 2024-02-19T22:07:59.217 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 777 /etc/ceph 2024-02-19T22:07:59.248 INFO:tasks.cephadm:Writing seed config... 2024-02-19T22:07:59.249 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-02-19T22:07:59.249 INFO:tasks.cephadm: override: [client] debug client = 20 2024-02-19T22:07:59.249 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-02-19T22:07:59.249 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-02-19T22:07:59.249 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-02-19T22:07:59.249 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-02-19T22:07:59.250 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-02-19T22:07:59.250 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-02-19T22:07:59.250 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-02-19T22:07:59.250 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-02-19T22:07:59.250 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-02-19T22:07:59.250 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-02-19T22:07:59.250 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-02-19T22:07:59.250 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-02-19T22:07:59.250 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-02-19T22:07:59.250 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-02-19T22:07:59.250 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-19T22:07:59.250 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-19T22:07:59.250 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-19T22:07:59.250 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-19T22:07:59.250 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-19T22:07:59.251 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-02-19T22:07:59.251 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-02-19T22:07:59.251 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-02-19T22:07:59.251 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-02-19T22:07:59.251 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-02-19T22:07:59.251 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-02-19T22:07:59.251 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-02-19T22:07:59.251 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-02-19T22:07:59.251 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-19T22:07:59.251 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-19T22:07:59.251 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-02-19T22:07:59.251 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-02-19T22:07:59.251 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-02-19T22:07:59.251 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-02-19T22:07:59.251 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-02-19T22:07:59.252 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-02-19T22:07:59.252 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-02-19T22:07:59.252 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-02-19T22:07:59.252 DEBUG:teuthology.orchestra.run.smithi052:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-19T22:07:59.279 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 56739a54-cf73-11ee-95bb-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-02-19T22:07:59.279 DEBUG:teuthology.orchestra.run.smithi052:mon.smithi052> sudo journalctl -f -n 0 -u ceph-56739a54-cf73-11ee-95bb-87774f69a715@mon.smithi052.service 2024-02-19T22:07:59.321 INFO:tasks.cephadm:Bootstrapping... 2024-02-19T22:07:59.321 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid 56739a54-cf73-11ee-95bb-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.52 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-19T22:07:59.351 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:-- Logs begin at Mon 2024-02-19 21:54:34 UTC. -- 2024-02-19T22:07:59.615 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-02-19T22:08:00.374 INFO:teuthology.orchestra.run.smithi052.stderr:/bin/podman: 4.9.3-dev 2024-02-19T22:08:00.386 INFO:teuthology.orchestra.run.smithi052.stderr:Verifying podman|docker is present... 2024-02-19T22:08:00.386 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-02-19T22:08:00.409 INFO:teuthology.orchestra.run.smithi052.stderr:/bin/podman: 4.9.3-dev 2024-02-19T22:08:00.430 INFO:teuthology.orchestra.run.smithi052.stderr:Verifying lvm2 is present... 2024-02-19T22:08:00.430 INFO:teuthology.orchestra.run.smithi052.stderr:Verifying time synchronization is in place... 2024-02-19T22:08:00.430 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: systemctl is-enabled chrony.service 2024-02-19T22:08:00.438 INFO:teuthology.orchestra.run.smithi052.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-19T22:08:00.438 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: systemctl is-active chrony.service 2024-02-19T22:08:00.444 INFO:teuthology.orchestra.run.smithi052.stderr:systemctl: inactive 2024-02-19T22:08:00.445 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: systemctl is-enabled chronyd.service 2024-02-19T22:08:00.451 INFO:teuthology.orchestra.run.smithi052.stderr:systemctl: enabled 2024-02-19T22:08:00.452 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: systemctl is-active chronyd.service 2024-02-19T22:08:00.457 INFO:teuthology.orchestra.run.smithi052.stderr:systemctl: active 2024-02-19T22:08:00.458 INFO:teuthology.orchestra.run.smithi052.stderr:Unit chronyd.service is enabled and running 2024-02-19T22:08:00.458 INFO:teuthology.orchestra.run.smithi052.stderr:Repeating the final host check... 2024-02-19T22:08:00.458 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-02-19T22:08:00.480 INFO:teuthology.orchestra.run.smithi052.stderr:/bin/podman: 4.9.3-dev 2024-02-19T22:08:00.494 INFO:teuthology.orchestra.run.smithi052.stderr:podman|docker (/bin/podman) is present 2024-02-19T22:08:00.495 INFO:teuthology.orchestra.run.smithi052.stderr:systemctl is present 2024-02-19T22:08:00.495 INFO:teuthology.orchestra.run.smithi052.stderr:lvcreate is present 2024-02-19T22:08:00.495 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: systemctl is-enabled chrony.service 2024-02-19T22:08:00.500 INFO:teuthology.orchestra.run.smithi052.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-19T22:08:00.501 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: systemctl is-active chrony.service 2024-02-19T22:08:00.506 INFO:teuthology.orchestra.run.smithi052.stderr:systemctl: inactive 2024-02-19T22:08:00.507 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: systemctl is-enabled chronyd.service 2024-02-19T22:08:00.513 INFO:teuthology.orchestra.run.smithi052.stderr:systemctl: enabled 2024-02-19T22:08:00.513 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: systemctl is-active chronyd.service 2024-02-19T22:08:00.518 INFO:teuthology.orchestra.run.smithi052.stderr:systemctl: active 2024-02-19T22:08:00.519 INFO:teuthology.orchestra.run.smithi052.stderr:Unit chronyd.service is enabled and running 2024-02-19T22:08:00.519 INFO:teuthology.orchestra.run.smithi052.stderr:Host looks OK 2024-02-19T22:08:00.520 INFO:teuthology.orchestra.run.smithi052.stderr:Cluster fsid: 56739a54-cf73-11ee-95bb-87774f69a715 2024-02-19T22:08:00.520 INFO:teuthology.orchestra.run.smithi052.stderr:Acquiring lock 140610839112168 on /run/cephadm/56739a54-cf73-11ee-95bb-87774f69a715.lock 2024-02-19T22:08:00.520 INFO:teuthology.orchestra.run.smithi052.stderr:Lock 140610839112168 acquired on /run/cephadm/56739a54-cf73-11ee-95bb-87774f69a715.lock 2024-02-19T22:08:00.520 INFO:teuthology.orchestra.run.smithi052.stderr:Verifying IP 172.21.15.52 port 3300 ... 2024-02-19T22:08:00.520 INFO:teuthology.orchestra.run.smithi052.stderr:Verifying IP 172.21.15.52 port 6789 ... 2024-02-19T22:08:00.521 INFO:teuthology.orchestra.run.smithi052.stderr:Base mon IP is 172.21.15.52, final addrv is [v2:172.21.15.52:3300,v1:172.21.15.52:6789] 2024-02-19T22:08:00.521 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /sbin/ip route ls 2024-02-19T22:08:00.523 INFO:teuthology.orchestra.run.smithi052.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.52 metric 100 2024-02-19T22:08:00.523 INFO:teuthology.orchestra.run.smithi052.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.52 metric 100 2024-02-19T22:08:00.524 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /sbin/ip -6 route ls 2024-02-19T22:08:00.526 INFO:teuthology.orchestra.run.smithi052.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-02-19T22:08:00.526 INFO:teuthology.orchestra.run.smithi052.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2024-02-19T22:08:00.526 INFO:teuthology.orchestra.run.smithi052.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1742sec hoplimit 64 pref medium 2024-02-19T22:08:00.526 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /sbin/ip -6 addr ls 2024-02-19T22:08:00.529 INFO:teuthology.orchestra.run.smithi052.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-19T22:08:00.529 INFO:teuthology.orchestra.run.smithi052.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-02-19T22:08:00.529 INFO:teuthology.orchestra.run.smithi052.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-19T22:08:00.529 INFO:teuthology.orchestra.run.smithi052.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2024-02-19T22:08:00.529 INFO:teuthology.orchestra.run.smithi052.stderr:/sbin/ip: inet6 fe80::ec4:7aff:febd:15de/64 scope link 2024-02-19T22:08:00.529 INFO:teuthology.orchestra.run.smithi052.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-19T22:08:00.530 INFO:teuthology.orchestra.run.smithi052.stderr:Mon IP 172.21.15.52 is in CIDR network 172.21.0.0/20 2024-02-19T22:08:00.530 INFO:teuthology.orchestra.run.smithi052.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-19T22:08:00.531 INFO:teuthology.orchestra.run.smithi052.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2024-02-19T22:08:00.532 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2024-02-19T22:08:00.625 INFO:teuthology.orchestra.run.smithi052.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2024-02-19T22:08:07.563 INFO:teuthology.orchestra.run.smithi052.stderr:/bin/podman: Getting image source signatures 2024-02-19T22:08:07.580 INFO:teuthology.orchestra.run.smithi052.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2024-02-19T22:08:07.580 INFO:teuthology.orchestra.run.smithi052.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2024-02-19T22:08:23.756 INFO:teuthology.orchestra.run.smithi052.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-02-19T22:08:23.931 INFO:teuthology.orchestra.run.smithi052.stderr:/bin/podman: Writing manifest to image destination 2024-02-19T22:08:24.214 INFO:teuthology.orchestra.run.smithi052.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-02-19T22:08:24.225 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi052 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2024-02-19T22:08:24.920 INFO:teuthology.orchestra.run.smithi052.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-02-19T22:08:25.529 INFO:teuthology.orchestra.run.smithi052.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-02-19T22:08:25.529 INFO:teuthology.orchestra.run.smithi052.stderr:Extracting ceph user uid/gid from container image... 2024-02-19T22:08:25.529 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi052 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2024-02-19T22:08:26.093 INFO:teuthology.orchestra.run.smithi052.stderr:stat: 167 167 2024-02-19T22:08:26.724 INFO:teuthology.orchestra.run.smithi052.stderr:Creating initial keys... 2024-02-19T22:08:26.724 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi052 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-02-19T22:08:27.248 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-authtool: AQBb0dNl3hWkCRAAFM21e5/69B+UwDej2ShtXg== 2024-02-19T22:08:27.811 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi052 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-02-19T22:08:28.321 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-authtool: AQBc0dNl7hs+DhAAnv/nJNFGIn9jw0PB2aeBNA== 2024-02-19T22:08:29.045 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi052 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-02-19T22:08:29.609 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-authtool: AQBd0dNl8HiTHhAAzK5C7hdwiJx/piA675CWMA== 2024-02-19T22:08:30.117 INFO:teuthology.orchestra.run.smithi052.stderr:Creating initial monmap... 2024-02-19T22:08:30.117 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi052 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmpv5x1es3d:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid 56739a54-cf73-11ee-95bb-87774f69a715 --addv smithi052 [v2:172.21.15.52:3300,v1:172.21.15.52:6789] /tmp/monmap 2024-02-19T22:08:30.732 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-19T22:08:30.732 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 56739a54-cf73-11ee-95bb-87774f69a715 2024-02-19T22:08:30.732 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-19T22:08:31.294 INFO:teuthology.orchestra.run.smithi052.stderr:monmaptool for smithi052 [v2:172.21.15.52:3300,v1:172.21.15.52:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-19T22:08:31.294 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/monmaptool: set fsid to 56739a54-cf73-11ee-95bb-87774f69a715 2024-02-19T22:08:31.294 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-19T22:08:31.294 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-02-19T22:08:31.294 INFO:teuthology.orchestra.run.smithi052.stderr:Creating mon... 2024-02-19T22:08:31.295 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi052 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/mon.smithi052:/var/lib/ceph/mon/ceph-smithi052:z -v /tmp/ceph-tmpidozjouz:/tmp/keyring:z -v /tmp/ceph-tmpv5x1es3d:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi052 --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2024-02-19T22:08:31.909 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.785+0000 7fe05af88700 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-19T22:08:31.909 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.786+0000 7fe05af88700 1 imported monmap: 2024-02-19T22:08:31.910 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: epoch 0 2024-02-19T22:08:31.910 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: fsid 56739a54-cf73-11ee-95bb-87774f69a715 2024-02-19T22:08:31.910 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: last_changed 2024-02-19T22:08:30.647471+0000 2024-02-19T22:08:31.910 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: created 2024-02-19T22:08:30.647471+0000 2024-02-19T22:08:31.910 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2024-02-19T22:08:31.910 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: election_strategy: 1 2024-02-19T22:08:31.910 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.smithi052 2024-02-19T22:08:31.910 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: 2024-02-19T22:08:31.911 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.786+0000 7fe05af88700 0 /usr/bin/ceph-mon: set fsid to 56739a54-cf73-11ee-95bb-87774f69a715 2024-02-19T22:08:31.911 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: RocksDB version: 6.8.1 2024-02-19T22:08:31.911 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: 2024-02-19T22:08:31.911 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-19T22:08:31.911 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Compile date May 13 2021 2024-02-19T22:08:31.911 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: DB SUMMARY 2024-02-19T22:08:31.911 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: 2024-02-19T22:08:31.911 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi052/store.db dir, Total Num: 0, files: 2024-02-19T22:08:31.912 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: 2024-02-19T22:08:31.912 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi052/store.db: 2024-02-19T22:08:31.912 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: 2024-02-19T22:08:31.912 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.error_if_exists: 0 2024-02-19T22:08:31.912 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.create_if_missing: 1 2024-02-19T22:08:31.912 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.paranoid_checks: 1 2024-02-19T22:08:31.912 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.env: 0x558c069231c0 2024-02-19T22:08:31.912 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.fs: Posix File System 2024-02-19T22:08:31.913 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.info_log: 0x558c0731eec0 2024-02-19T22:08:31.913 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-19T22:08:31.913 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.statistics: (nil) 2024-02-19T22:08:31.913 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.use_fsync: 0 2024-02-19T22:08:31.913 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.max_log_file_size: 0 2024-02-19T22:08:31.913 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-19T22:08:31.913 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-19T22:08:31.913 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-19T22:08:31.913 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-19T22:08:31.914 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.allow_fallocate: 1 2024-02-19T22:08:31.914 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-19T22:08:31.914 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-19T22:08:31.914 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.use_direct_reads: 0 2024-02-19T22:08:31.914 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-19T22:08:31.914 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.create_missing_column_families: 0 2024-02-19T22:08:31.914 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.db_log_dir: 2024-02-19T22:08:31.914 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi052/store.db 2024-02-19T22:08:31.915 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-19T22:08:31.915 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.max_subcompactions: 1 2024-02-19T22:08:31.915 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.max_background_flushes: -1 2024-02-19T22:08:31.915 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-19T22:08:31.915 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-19T22:08:31.915 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-19T22:08:31.915 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-19T22:08:31.915 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-19T22:08:31.915 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.advise_random_on_open: 1 2024-02-19T22:08:31.916 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-19T22:08:31.916 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.write_buffer_manager: 0x558c07325530 2024-02-19T22:08:31.916 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-19T22:08:31.916 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-19T22:08:31.916 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-19T22:08:31.916 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-19T22:08:31.916 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.rate_limiter: (nil) 2024-02-19T22:08:31.916 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-19T22:08:31.917 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-19T22:08:31.917 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-19T22:08:31.917 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-19T22:08:31.917 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.unordered_write: 0 2024-02-19T22:08:31.917 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-19T22:08:31.917 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-19T22:08:31.917 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-19T22:08:31.917 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-19T22:08:31.918 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.row_cache: None 2024-02-19T22:08:31.918 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.wal_filter: None 2024-02-19T22:08:31.918 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-19T22:08:31.918 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-19T22:08:31.918 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.preserve_deletes: 0 2024-02-19T22:08:31.918 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.two_write_queues: 0 2024-02-19T22:08:31.918 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.manual_wal_flush: 0 2024-02-19T22:08:31.918 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.atomic_flush: 0 2024-02-19T22:08:31.919 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-19T22:08:31.919 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-19T22:08:31.919 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-19T22:08:31.919 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.log_readahead_size: 0 2024-02-19T22:08:31.919 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-19T22:08:31.919 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.max_background_jobs: 2 2024-02-19T22:08:31.919 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.max_background_compactions: -1 2024-02-19T22:08:31.919 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-19T22:08:31.919 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-19T22:08:31.920 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-19T22:08:31.920 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.max_total_wal_size: 0 2024-02-19T22:08:31.920 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-19T22:08:31.920 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-19T22:08:31.920 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-19T22:08:31.920 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-19T22:08:31.920 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.max_open_files: -1 2024-02-19T22:08:31.920 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.bytes_per_sync: 0 2024-02-19T22:08:31.921 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-19T22:08:31.921 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-19T22:08:31.921 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-19T22:08:31.921 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Compression algorithms supported: 2024-02-19T22:08:31.921 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-19T22:08:31.921 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: kZSTD supported: 0 2024-02-19T22:08:31.921 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: kXpressCompression supported: 0 2024-02-19T22:08:31.921 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-19T22:08:31.922 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: kLZ4Compression supported: 1 2024-02-19T22:08:31.922 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: kBZip2Compression supported: 0 2024-02-19T22:08:31.922 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: kZlibCompression supported: 1 2024-02-19T22:08:31.922 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: kSnappyCompression supported: 1 2024-02-19T22:08:31.922 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-19T22:08:31.922 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.787+0000 7fe05af88700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2024-02-19T22:08:31.922 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: 2024-02-19T22:08:31.923 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi052/store.db/MANIFEST-000001 2024-02-19T22:08:31.923 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: 2024-02-19T22:08:31.923 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-19T22:08:31.923 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: 2024-02-19T22:08:31.923 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-19T22:08:31.923 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.merge_operator: 2024-02-19T22:08:31.923 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.compaction_filter: None 2024-02-19T22:08:31.923 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.compaction_filter_factory: None 2024-02-19T22:08:31.923 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-19T22:08:31.924 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-19T22:08:31.924 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x558c07237cb8) 2024-02-19T22:08:31.924 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2024-02-19T22:08:31.924 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-19T22:08:31.924 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-19T22:08:31.924 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2024-02-19T22:08:31.924 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: index_type: 0 2024-02-19T22:08:31.924 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2024-02-19T22:08:31.925 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: index_shortening: 1 2024-02-19T22:08:31.925 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2024-02-19T22:08:31.925 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2024-02-19T22:08:31.925 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: checksum: 1 2024-02-19T22:08:31.925 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2024-02-19T22:08:31.925 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: block_cache: 0x558c0726cd10 2024-02-19T22:08:31.925 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2024-02-19T22:08:31.925 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: block_cache_options: 2024-02-19T22:08:31.926 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: capacity : 536870912 2024-02-19T22:08:31.926 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2024-02-19T22:08:31.926 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2024-02-19T22:08:31.926 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2024-02-19T22:08:31.926 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2024-02-19T22:08:31.926 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2024-02-19T22:08:31.926 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: block_size: 4096 2024-02-19T22:08:31.926 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2024-02-19T22:08:31.927 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2024-02-19T22:08:31.927 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2024-02-19T22:08:31.927 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2024-02-19T22:08:31.927 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: partition_filters: 0 2024-02-19T22:08:31.927 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2024-02-19T22:08:31.927 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-19T22:08:31.927 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2024-02-19T22:08:31.927 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: verify_compression: 0 2024-02-19T22:08:31.927 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2024-02-19T22:08:31.928 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: format_version: 2 2024-02-19T22:08:31.928 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2024-02-19T22:08:31.928 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: block_align: 0 2024-02-19T22:08:31.928 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: 2024-02-19T22:08:31.928 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-19T22:08:31.928 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-19T22:08:31.928 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.compression: NoCompression 2024-02-19T22:08:31.928 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-19T22:08:31.929 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-19T22:08:31.929 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-19T22:08:31.929 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.num_levels: 7 2024-02-19T22:08:31.929 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-19T22:08:31.929 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-19T22:08:31.929 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-19T22:08:31.929 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-19T22:08:31.929 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-19T22:08:31.930 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-19T22:08:31.930 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-19T22:08:31.930 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-19T22:08:31.930 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-19T22:08:31.930 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-19T22:08:31.930 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.compression_opts.level: 32767 2024-02-19T22:08:31.930 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-19T22:08:31.930 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-19T22:08:31.930 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-19T22:08:31.931 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.compression_opts.enabled: false 2024-02-19T22:08:31.931 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-19T22:08:31.931 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-19T22:08:31.931 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-19T22:08:31.931 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-19T22:08:31.931 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-19T22:08:31.931 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-19T22:08:31.931 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-19T22:08:31.932 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-19T22:08:31.932 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-19T22:08:31.932 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-19T22:08:31.932 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-19T22:08:31.932 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-19T22:08:31.932 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-19T22:08:31.932 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-19T22:08:31.932 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-19T22:08:31.933 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-19T22:08:31.933 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-19T22:08:31.933 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.arena_block_size: 4194304 2024-02-19T22:08:31.933 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-19T22:08:31.933 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-19T22:08:31.933 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-19T22:08:31.933 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-19T22:08:31.933 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-19T22:08:31.933 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-19T22:08:31.934 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-19T22:08:31.934 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-19T22:08:31.934 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-19T22:08:31.934 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-19T22:08:31.934 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-19T22:08:31.934 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-19T22:08:31.935 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-19T22:08:31.935 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-19T22:08:31.935 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.table_properties_collectors: 2024-02-19T22:08:31.935 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.inplace_update_support: 0 2024-02-19T22:08:31.935 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-19T22:08:31.935 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-19T22:08:31.935 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-19T22:08:31.936 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-19T22:08:31.936 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.bloom_locality: 0 2024-02-19T22:08:31.936 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.max_successive_merges: 0 2024-02-19T22:08:31.936 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-19T22:08:31.936 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-19T22:08:31.936 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.force_consistency_checks: 0 2024-02-19T22:08:31.936 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-19T22:08:31.936 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.ttl: 2592000 2024-02-19T22:08:31.937 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-19T22:08:31.937 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi052/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-19T22:08:31.937 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: 2024-02-19T22:08:31.937 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.788+0000 7fe05af88700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-02-19T22:08:31.937 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: 2024-02-19T22:08:31.937 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.789+0000 7fe05af88700 4 rocksdb: DB pointer 0x558c07331800 2024-02-19T22:08:31.937 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.789+0000 7fe043e21700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-19T22:08:31.937 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.789+0000 7fe043e21700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-19T22:08:31.938 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2024-02-19T22:08:31.938 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T22:08:31.938 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-19T22:08:31.938 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-19T22:08:31.938 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T22:08:31.938 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-19T22:08:31.938 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-19T22:08:31.938 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T22:08:31.939 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: 2024-02-19T22:08:31.939 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-19T22:08:31.939 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T22:08:31.939 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T22:08:31.939 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-19T22:08:31.939 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-19T22:08:31.939 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: 2024-02-19T22:08:31.939 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-19T22:08:31.940 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T22:08:31.940 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T22:08:31.940 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T22:08:31.940 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-02-19T22:08:31.940 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-19T22:08:31.940 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-02-19T22:08:31.940 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-02-19T22:08:31.941 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-02-19T22:08:31.941 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T22:08:31.941 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T22:08:31.941 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-19T22:08:31.941 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: 2024-02-19T22:08:31.941 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-02-19T22:08:31.941 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: 2024-02-19T22:08:31.941 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-19T22:08:31.942 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T22:08:31.942 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T22:08:31.942 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-19T22:08:31.942 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-19T22:08:31.942 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: 2024-02-19T22:08:31.942 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-19T22:08:31.942 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T22:08:31.943 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T22:08:31.943 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T22:08:31.943 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-02-19T22:08:31.943 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-19T22:08:31.943 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-02-19T22:08:31.943 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-02-19T22:08:31.943 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-02-19T22:08:31.943 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T22:08:31.944 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T22:08:31.944 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-19T22:08:31.944 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: 2024-02-19T22:08:31.944 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-02-19T22:08:31.944 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: 2024-02-19T22:08:31.944 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.790+0000 7fe05af88700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2024-02-19T22:08:31.945 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.791+0000 7fe05af88700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2024-02-19T22:08:31.945 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:08:31.791+0000 7fe05af88700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi052 for mon.smithi052 2024-02-19T22:08:32.515 INFO:teuthology.orchestra.run.smithi052.stderr:create mon.smithi052 on 2024-02-19T22:08:32.515 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/56739a54-cf73-11ee-95bb-87774f69a715 2024-02-19T22:08:32.521 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: systemctl disable ceph.target 2024-02-19T22:08:32.530 INFO:teuthology.orchestra.run.smithi052.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-02-19T22:08:32.626 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: systemctl enable ceph.target 2024-02-19T22:08:32.633 INFO:teuthology.orchestra.run.smithi052.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-19T22:08:32.719 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: systemctl start ceph.target 2024-02-19T22:08:32.726 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: systemctl enable ceph-56739a54-cf73-11ee-95bb-87774f69a715.target 2024-02-19T22:08:32.732 INFO:teuthology.orchestra.run.smithi052.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-56739a54-cf73-11ee-95bb-87774f69a715.target → /etc/systemd/system/ceph-56739a54-cf73-11ee-95bb-87774f69a715.target. 2024-02-19T22:08:32.732 INFO:teuthology.orchestra.run.smithi052.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-56739a54-cf73-11ee-95bb-87774f69a715.target → /etc/systemd/system/ceph-56739a54-cf73-11ee-95bb-87774f69a715.target. 2024-02-19T22:08:32.815 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: systemctl start ceph-56739a54-cf73-11ee-95bb-87774f69a715.target 2024-02-19T22:08:32.822 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: systemctl daemon-reload 2024-02-19T22:08:32.914 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: systemctl stop ceph-56739a54-cf73-11ee-95bb-87774f69a715@mon.smithi052 2024-02-19T22:08:32.941 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: systemctl reset-failed ceph-56739a54-cf73-11ee-95bb-87774f69a715@mon.smithi052 2024-02-19T22:08:32.947 INFO:teuthology.orchestra.run.smithi052.stderr:systemctl: Failed to reset failed state of unit ceph-56739a54-cf73-11ee-95bb-87774f69a715@mon.smithi052.service: Unit ceph-56739a54-cf73-11ee-95bb-87774f69a715@mon.smithi052.service not loaded. 2024-02-19T22:08:32.947 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: systemctl enable ceph-56739a54-cf73-11ee-95bb-87774f69a715@mon.smithi052 2024-02-19T22:08:32.954 INFO:teuthology.orchestra.run.smithi052.stderr:systemctl: Created symlink /etc/systemd/system/ceph-56739a54-cf73-11ee-95bb-87774f69a715.target.wants/ceph-56739a54-cf73-11ee-95bb-87774f69a715@mon.smithi052.service → /etc/systemd/system/ceph-56739a54-cf73-11ee-95bb-87774f69a715@.service. 2024-02-19T22:08:33.044 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: systemctl start ceph-56739a54-cf73-11ee-95bb-87774f69a715@mon.smithi052 2024-02-19T22:08:33.092 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:33 smithi052 systemd[1]: Starting Ceph mon.smithi052 for 56739a54-cf73-11ee-95bb-87774f69a715... 2024-02-19T22:08:33.624 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-19T22:08:33.632 INFO:teuthology.orchestra.run.smithi052.stderr:systemctl: disabled 2024-02-19T22:08:33.633 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: systemctl is-active firewalld.service 2024-02-19T22:08:33.640 INFO:teuthology.orchestra.run.smithi052.stderr:systemctl: inactive 2024-02-19T22:08:33.641 INFO:teuthology.orchestra.run.smithi052.stderr:firewalld.service is not enabled 2024-02-19T22:08:33.641 INFO:teuthology.orchestra.run.smithi052.stderr:Not possible to enable service . firewalld.service is not available 2024-02-19T22:08:33.641 INFO:teuthology.orchestra.run.smithi052.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-02-19T22:08:33.642 INFO:teuthology.orchestra.run.smithi052.stderr:Waiting for mon to start... 2024-02-19T22:08:33.642 INFO:teuthology.orchestra.run.smithi052.stderr:Waiting for mon... 2024-02-19T22:08:33.642 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi052 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/mon.smithi052:/var/lib/ceph/mon/ceph-smithi052:z -v /tmp/ceph-tmphvym4zi3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp6ou7w4ul:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2024-02-19T22:08:33.842 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:33 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[37748]: cluster 2024-02-19T22:08:33.546175+0000 mon.smithi052 (mon.0) 0 : [INF] mkfs 56739a54-cf73-11ee-95bb-87774f69a715 2024-02-19T22:08:33.842 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:33 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[37748]: cluster 2024-02-19T22:08:33.531000+0000 mon.smithi052 (mon.0) 1 : cluster [INF] mon.smithi052 is new leader, mons smithi052 in quorum (ranks 0) 2024-02-19T22:08:33.842 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:33 smithi052 bash[37733]: 64c0682736409feed136c8206fb6ecddd02f9b5bb401c426b7ac0599b85c35f2 2024-02-19T22:08:33.842 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:33 smithi052 systemd[1]: Started Ceph mon.smithi052 for 56739a54-cf73-11ee-95bb-87774f69a715. 2024-02-19T22:08:34.184 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.158+0000 7f18a132b700 1 Processor -- start 2024-02-19T22:08:34.185 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.158+0000 7f18a132b700 1 -- start start 2024-02-19T22:08:34.185 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.158+0000 7f18a132b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f189c0703e0 0x7f189c0febc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:34.185 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.158+0000 7f18a132b700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f189c070800 con 0x7f189c0703e0 2024-02-19T22:08:34.185 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.158+0000 7f189affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f189c0703e0 0x7f189c0febc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:34.185 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.158+0000 7f189affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f189c0703e0 0x7f189c0febc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:42380/0 (socket says 172.21.15.52:42380) 2024-02-19T22:08:34.185 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.158+0000 7f189affd700 1 -- 172.21.15.52:0/3458416047 learned_addr learned my addr 172.21.15.52:0/3458416047 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:08:34.185 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.159+0000 7f189affd700 1 -- 172.21.15.52:0/3458416047 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f189c0ff0b0 con 0x7f189c0703e0 2024-02-19T22:08:34.185 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.159+0000 7f189affd700 1 --2- 172.21.15.52:0/3458416047 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f189c0703e0 0x7f189c0febc0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f188c00b5f0 tx=0x7f188c006ee0).ready entity=mon.0 client_cookie=c9d169a07f9a51b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:34.185 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.159+0000 7f1899ffb700 1 -- 172.21.15.52:0/3458416047 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f188c00d600 con 0x7f189c0703e0 2024-02-19T22:08:34.185 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.159+0000 7f1899ffb700 1 -- 172.21.15.52:0/3458416047 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f188c00d760 con 0x7f189c0703e0 2024-02-19T22:08:34.185 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.159+0000 7f1899ffb700 1 -- 172.21.15.52:0/3458416047 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f188c00da30 con 0x7f189c0703e0 2024-02-19T22:08:34.185 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.159+0000 7f18a132b700 1 -- 172.21.15.52:0/3458416047 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f189c0703e0 msgr2=0x7f189c0febc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:34.185 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.159+0000 7f18a132b700 1 --2- 172.21.15.52:0/3458416047 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f189c0703e0 0x7f189c0febc0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f188c00b5f0 tx=0x7f188c006ee0).stop 2024-02-19T22:08:34.186 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.160+0000 7f18a132b700 1 -- 172.21.15.52:0/3458416047 shutdown_connections 2024-02-19T22:08:34.186 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.160+0000 7f18a132b700 1 --2- 172.21.15.52:0/3458416047 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f189c0703e0 0x7f189c0febc0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:34.186 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.160+0000 7f18a132b700 1 -- 172.21.15.52:0/3458416047 >> 172.21.15.52:0/3458416047 conn(0x7f189c0f65c0 msgr2=0x7f189c0f89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:34.186 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.160+0000 7f18a132b700 1 -- 172.21.15.52:0/3458416047 shutdown_connections 2024-02-19T22:08:34.186 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.160+0000 7f18a132b700 1 -- 172.21.15.52:0/3458416047 wait complete. 2024-02-19T22:08:34.186 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.160+0000 7f18a132b700 1 Processor -- start 2024-02-19T22:08:34.186 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.160+0000 7f18a132b700 1 -- start start 2024-02-19T22:08:34.186 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.160+0000 7f18a132b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f189c0703e0 0x7f189c10a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:34.186 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.160+0000 7f18a132b700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f189c10a8b0 con 0x7f189c0703e0 2024-02-19T22:08:34.186 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.161+0000 7f189affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f189c0703e0 0x7f189c10a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:34.186 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.161+0000 7f189affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f189c0703e0 0x7f189c10a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:42384/0 (socket says 172.21.15.52:42384) 2024-02-19T22:08:34.186 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.161+0000 7f189affd700 1 -- 172.21.15.52:0/405167727 learned_addr learned my addr 172.21.15.52:0/405167727 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:08:34.186 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.161+0000 7f189affd700 1 -- 172.21.15.52:0/405167727 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f188c00b200 con 0x7f189c0703e0 2024-02-19T22:08:34.187 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.161+0000 7f189affd700 1 --2- 172.21.15.52:0/405167727 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f189c0703e0 0x7f189c10a3c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f188c006c70 tx=0x7f188c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:34.187 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.161+0000 7f18837fe700 1 -- 172.21.15.52:0/405167727 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f188c011b40 con 0x7f189c0703e0 2024-02-19T22:08:34.187 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.161+0000 7f18837fe700 1 -- 172.21.15.52:0/405167727 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f188c011ca0 con 0x7f189c0703e0 2024-02-19T22:08:34.187 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.161+0000 7f18837fe700 1 -- 172.21.15.52:0/405167727 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f188c01d430 con 0x7f189c0703e0 2024-02-19T22:08:34.187 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.161+0000 7f18a132b700 1 -- 172.21.15.52:0/405167727 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f189c10aab0 con 0x7f189c0703e0 2024-02-19T22:08:34.187 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.161+0000 7f18a132b700 1 -- 172.21.15.52:0/405167727 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f189c10af40 con 0x7f189c0703e0 2024-02-19T22:08:34.187 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.161+0000 7f18837fe700 1 -- 172.21.15.52:0/405167727 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f188c00f070 con 0x7f189c0703e0 2024-02-19T22:08:34.187 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.161+0000 7f18837fe700 1 -- 172.21.15.52:0/405167727 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f188c027460 con 0x7f189c0703e0 2024-02-19T22:08:34.187 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.162+0000 7f18a132b700 1 -- 172.21.15.52:0/405167727 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f189c045ad0 con 0x7f189c0703e0 2024-02-19T22:08:34.187 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.164+0000 7f18837fe700 1 -- 172.21.15.52:0/405167727 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f188c016070 con 0x7f189c0703e0 2024-02-19T22:08:34.238 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.234+0000 7f18a132b700 1 -- 172.21.15.52:0/405167727 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f189c1040a0 con 0x7f189c0703e0 2024-02-19T22:08:34.238 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.235+0000 7f18837fe700 1 -- 172.21.15.52:0/405167727 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f188c036070 con 0x7f189c0703e0 2024-02-19T22:08:34.238 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: cluster: 2024-02-19T22:08:34.238 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: id: 56739a54-cf73-11ee-95bb-87774f69a715 2024-02-19T22:08:34.238 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-02-19T22:08:34.238 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.239 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: services: 2024-02-19T22:08:34.239 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi052 (age 0.690701s) 2024-02-19T22:08:34.239 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: mgr: no daemons active 2024-02-19T22:08:34.239 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-02-19T22:08:34.239 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.239 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: data: 2024-02-19T22:08:34.239 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-02-19T22:08:34.239 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-02-19T22:08:34.239 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-02-19T22:08:34.239 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: pgs: 2024-02-19T22:08:34.239 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.239 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.235+0000 7f18a132b700 1 -- 172.21.15.52:0/405167727 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f189c0703e0 msgr2=0x7f189c10a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:34.239 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.235+0000 7f18a132b700 1 --2- 172.21.15.52:0/405167727 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f189c0703e0 0x7f189c10a3c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f188c006c70 tx=0x7f188c016000).stop 2024-02-19T22:08:34.239 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.236+0000 7f18a132b700 1 -- 172.21.15.52:0/405167727 shutdown_connections 2024-02-19T22:08:34.239 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.236+0000 7f18a132b700 1 --2- 172.21.15.52:0/405167727 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f189c0703e0 0x7f189c10a3c0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:34.240 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.236+0000 7f18a132b700 1 -- 172.21.15.52:0/405167727 >> 172.21.15.52:0/405167727 conn(0x7f189c0f65c0 msgr2=0x7f189c0f8250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:34.240 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.236+0000 7f18a132b700 1 -- 172.21.15.52:0/405167727 shutdown_connections 2024-02-19T22:08:34.240 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.236+0000 7f18a132b700 1 -- 172.21.15.52:0/405167727 wait complete. 2024-02-19T22:08:34.842 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:34 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[37748]: cluster 2024-02-19T22:08:33.545750+0000 mon.smithi052 (mon.0) 2 : cluster [INF] mon.smithi052 is new leader, mons smithi052 in quorum (ranks 0) 2024-02-19T22:08:34.842 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:34 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[37748]: cluster 2024-02-19T22:08:33.545874+0000 mon.smithi052 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi052=[v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0]} 2024-02-19T22:08:34.842 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:34 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[37748]: cluster 2024-02-19T22:08:33.547213+0000 mon.smithi052 (mon.0) 4 : cluster [DBG] fsmap 2024-02-19T22:08:34.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:34 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[37748]: cluster 2024-02-19T22:08:33.548922+0000 mon.smithi052 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-02-19T22:08:34.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:34 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[37748]: cluster 2024-02-19T22:08:33.549271+0000 mon.smithi052 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2024-02-19T22:08:34.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:34 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[37748]: audit 2024-02-19T22:08:34.236421+0000 mon.smithi052 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.52:0/405167727' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-19T22:08:34.901 INFO:teuthology.orchestra.run.smithi052.stderr:mon is available 2024-02-19T22:08:34.901 INFO:teuthology.orchestra.run.smithi052.stderr:Assimilating anything we can from ceph.conf... 2024-02-19T22:08:34.901 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi052 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphvym4zi3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp6ou7w4ul:/etc/ceph/ceph.conf:z -v /var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/mon.smithi052:/var/lib/ceph/mon/ceph-smithi052:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi052/config 2024-02-19T22:08:35.402 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.400+0000 7f1fe8c1a700 1 Processor -- start 2024-02-19T22:08:35.403 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.400+0000 7f1fe8c1a700 1 -- start start 2024-02-19T22:08:35.403 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.401+0000 7f1fe8c1a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1fe40703d0 0x7f1fe4100eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:35.403 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.401+0000 7f1fe8c1a700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe40fc8a0 con 0x7f1fe40703d0 2024-02-19T22:08:35.403 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.401+0000 7f1fe259c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1fe40703d0 0x7f1fe4100eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:35.403 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.401+0000 7f1fe259c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1fe40703d0 0x7f1fe4100eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:42392/0 (socket says 172.21.15.52:42392) 2024-02-19T22:08:35.403 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.401+0000 7f1fe259c700 1 -- 172.21.15.52:0/404264710 learned_addr learned my addr 172.21.15.52:0/404264710 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:08:35.403 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.401+0000 7f1fe259c700 1 -- 172.21.15.52:0/404264710 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe41013a0 con 0x7f1fe40703d0 2024-02-19T22:08:35.403 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.402+0000 7f1fe259c700 1 --2- 172.21.15.52:0/404264710 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1fe40703d0 0x7f1fe4100eb0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f1fcc00b5f0 tx=0x7f1fcc006ee0).ready entity=mon.0 client_cookie=151817b5ed1953e6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:35.403 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.402+0000 7f1fe159a700 1 -- 172.21.15.52:0/404264710 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fcc00d600 con 0x7f1fe40703d0 2024-02-19T22:08:35.403 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.402+0000 7f1fe159a700 1 -- 172.21.15.52:0/404264710 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fcc00d760 con 0x7f1fe40703d0 2024-02-19T22:08:35.403 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.402+0000 7f1fe159a700 1 -- 172.21.15.52:0/404264710 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fcc00da30 con 0x7f1fe40703d0 2024-02-19T22:08:35.405 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.402+0000 7f1fe8c1a700 1 -- 172.21.15.52:0/404264710 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1fe40703d0 msgr2=0x7f1fe4100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:35.405 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.402+0000 7f1fe8c1a700 1 --2- 172.21.15.52:0/404264710 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1fe40703d0 0x7f1fe4100eb0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f1fcc00b5f0 tx=0x7f1fcc006ee0).stop 2024-02-19T22:08:35.405 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.402+0000 7f1fe8c1a700 1 -- 172.21.15.52:0/404264710 shutdown_connections 2024-02-19T22:08:35.405 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.402+0000 7f1fe8c1a700 1 --2- 172.21.15.52:0/404264710 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1fe40703d0 0x7f1fe4100eb0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:35.405 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.402+0000 7f1fe8c1a700 1 -- 172.21.15.52:0/404264710 >> 172.21.15.52:0/404264710 conn(0x7f1fe40f6530 msgr2=0x7f1fe40f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:35.406 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.402+0000 7f1fe8c1a700 1 -- 172.21.15.52:0/404264710 shutdown_connections 2024-02-19T22:08:35.406 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.402+0000 7f1fe8c1a700 1 -- 172.21.15.52:0/404264710 wait complete. 2024-02-19T22:08:35.406 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.403+0000 7f1fe8c1a700 1 Processor -- start 2024-02-19T22:08:35.406 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.403+0000 7f1fe8c1a700 1 -- start start 2024-02-19T22:08:35.406 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.403+0000 7f1fe8c1a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1fe40703d0 0x7f1fe410c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:35.406 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.403+0000 7f1fe8c1a700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe410ca40 con 0x7f1fe40703d0 2024-02-19T22:08:35.406 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.404+0000 7f1fe259c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1fe40703d0 0x7f1fe410c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:35.406 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.404+0000 7f1fe259c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1fe40703d0 0x7f1fe410c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:42400/0 (socket says 172.21.15.52:42400) 2024-02-19T22:08:35.406 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.404+0000 7f1fe259c700 1 -- 172.21.15.52:0/1350257156 learned_addr learned my addr 172.21.15.52:0/1350257156 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:08:35.406 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.404+0000 7f1fe259c700 1 -- 172.21.15.52:0/1350257156 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fcc00b200 con 0x7f1fe40703d0 2024-02-19T22:08:35.407 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.404+0000 7f1fe259c700 1 --2- 172.21.15.52:0/1350257156 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1fe40703d0 0x7f1fe410c550 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f1fcc006c70 tx=0x7f1fcc016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:35.407 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.404+0000 7f1fdaffd700 1 -- 172.21.15.52:0/1350257156 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fcc011b40 con 0x7f1fe40703d0 2024-02-19T22:08:35.407 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.404+0000 7f1fdaffd700 1 -- 172.21.15.52:0/1350257156 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fcc011ca0 con 0x7f1fe40703d0 2024-02-19T22:08:35.407 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.404+0000 7f1fdaffd700 1 -- 172.21.15.52:0/1350257156 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fcc01d430 con 0x7f1fe40703d0 2024-02-19T22:08:35.407 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.404+0000 7f1fe8c1a700 1 -- 172.21.15.52:0/1350257156 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fe410cc40 con 0x7f1fe40703d0 2024-02-19T22:08:35.407 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.404+0000 7f1fe8c1a700 1 -- 172.21.15.52:0/1350257156 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fe410d0d0 con 0x7f1fe40703d0 2024-02-19T22:08:35.407 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.404+0000 7f1fdaffd700 1 -- 172.21.15.52:0/1350257156 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f1fcc00f070 con 0x7f1fe40703d0 2024-02-19T22:08:35.407 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.404+0000 7f1fdaffd700 1 -- 172.21.15.52:0/1350257156 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f1fcc027460 con 0x7f1fe40703d0 2024-02-19T22:08:35.407 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.405+0000 7f1fe8c1a700 1 -- 172.21.15.52:0/1350257156 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fe4045ad0 con 0x7f1fe40703d0 2024-02-19T22:08:35.408 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.408+0000 7f1fdaffd700 1 -- 172.21.15.52:0/1350257156 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f1fcc016070 con 0x7f1fe40703d0 2024-02-19T22:08:35.475 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.475+0000 7f1fe8c1a700 1 -- 172.21.15.52:0/1350257156 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f1fe41061f0 con 0x7f1fe40703d0 2024-02-19T22:08:35.481 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.478+0000 7f1fdaffd700 1 -- 172.21.15.52:0/1350257156 <== mon.0 v2:172.21.15.52:3300/0 7 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f1fcc01d660 con 0x7f1fe40703d0 2024-02-19T22:08:35.481 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.478+0000 7f1fdaffd700 1 -- 172.21.15.52:0/1350257156 <== mon.0 v2:172.21.15.52:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+489 (secure 0 0 0) 0x7f1fcc036020 con 0x7f1fe40703d0 2024-02-19T22:08:35.481 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.482 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: [global] 2024-02-19T22:08:35.482 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: fsid = 56739a54-cf73-11ee-95bb-87774f69a715 2024-02-19T22:08:35.482 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: log_to_journald = false 2024-02-19T22:08:35.482 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.52:3300,v1:172.21.15.52:6789] 2024-02-19T22:08:35.482 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-02-19T22:08:35.482 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-02-19T22:08:35.482 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-02-19T22:08:35.482 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-02-19T22:08:35.482 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.482 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: [mgr] 2024-02-19T22:08:35.482 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-02-19T22:08:35.482 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.482 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: [osd] 2024-02-19T22:08:35.482 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2024-02-19T22:08:35.482 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2024-02-19T22:08:35.483 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2024-02-19T22:08:35.483 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-02-19T22:08:35.483 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2024-02-19T22:08:35.483 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-02-19T22:08:35.483 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.479+0000 7f1fe8c1a700 1 -- 172.21.15.52:0/1350257156 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1fe40703d0 msgr2=0x7f1fe410c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:35.483 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.479+0000 7f1fe8c1a700 1 --2- 172.21.15.52:0/1350257156 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1fe40703d0 0x7f1fe410c550 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f1fcc006c70 tx=0x7f1fcc016000).stop 2024-02-19T22:08:35.483 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.479+0000 7f1fe8c1a700 1 -- 172.21.15.52:0/1350257156 shutdown_connections 2024-02-19T22:08:35.483 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.479+0000 7f1fe8c1a700 1 --2- 172.21.15.52:0/1350257156 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1fe40703d0 0x7f1fe410c550 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:35.483 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.479+0000 7f1fe8c1a700 1 -- 172.21.15.52:0/1350257156 >> 172.21.15.52:0/1350257156 conn(0x7f1fe40f6530 msgr2=0x7f1fe40f81f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:35.483 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.479+0000 7f1fe8c1a700 1 -- 172.21.15.52:0/1350257156 shutdown_connections 2024-02-19T22:08:35.483 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.479+0000 7f1fe8c1a700 1 -- 172.21.15.52:0/1350257156 wait complete. 2024-02-19T22:08:35.842 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:35 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[37748]: audit 2024-02-19T22:08:35.476451+0000 mon.smithi052 (mon.0) 8 : audit [INF] from='client.? 172.21.15.52:0/1350257156' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-19T22:08:35.842 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:35 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[37748]: audit 2024-02-19T22:08:35.479550+0000 mon.smithi052 (mon.0) 9 : audit [INF] from='client.? 172.21.15.52:0/1350257156' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-19T22:08:36.168 INFO:teuthology.orchestra.run.smithi052.stderr:Generating new minimal ceph.conf... 2024-02-19T22:08:36.169 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi052 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphvym4zi3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp6ou7w4ul:/etc/ceph/ceph.conf:z -v /var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/mon.smithi052:/var/lib/ceph/mon/ceph-smithi052:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi052/config 2024-02-19T22:08:36.735 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.732+0000 7f336f0fd700 1 Processor -- start 2024-02-19T22:08:36.735 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.732+0000 7f336f0fd700 1 -- start start 2024-02-19T22:08:36.735 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.732+0000 7f336f0fd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f33680703d0 0x7f33680feab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:36.735 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.732+0000 7f336f0fd700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33680fefa0 con 0x7f33680703d0 2024-02-19T22:08:36.735 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.732+0000 7f336ce99700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f33680703d0 0x7f33680feab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:36.735 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.732+0000 7f336ce99700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f33680703d0 0x7f33680feab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:42410/0 (socket says 172.21.15.52:42410) 2024-02-19T22:08:36.736 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.732+0000 7f336ce99700 1 -- 172.21.15.52:0/2664727378 learned_addr learned my addr 172.21.15.52:0/2664727378 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:08:36.736 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.732+0000 7f336ce99700 1 -- 172.21.15.52:0/2664727378 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33680ff0e0 con 0x7f33680703d0 2024-02-19T22:08:36.736 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.733+0000 7f336ce99700 1 --2- 172.21.15.52:0/2664727378 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f33680703d0 0x7f33680feab0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f335c00b5f0 tx=0x7f335c006ee0).ready entity=mon.0 client_cookie=c9fa3e467037738c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:36.736 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.733+0000 7f33677fe700 1 -- 172.21.15.52:0/2664727378 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f335c00d600 con 0x7f33680703d0 2024-02-19T22:08:36.736 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.733+0000 7f33677fe700 1 -- 172.21.15.52:0/2664727378 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f335c00dbf0 con 0x7f33680703d0 2024-02-19T22:08:36.736 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.733+0000 7f33677fe700 1 -- 172.21.15.52:0/2664727378 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f335c011cb0 con 0x7f33680703d0 2024-02-19T22:08:36.736 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.733+0000 7f336f0fd700 1 -- 172.21.15.52:0/2664727378 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f33680703d0 msgr2=0x7f33680feab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:36.736 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.733+0000 7f336f0fd700 1 --2- 172.21.15.52:0/2664727378 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f33680703d0 0x7f33680feab0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f335c00b5f0 tx=0x7f335c006ee0).stop 2024-02-19T22:08:36.736 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.733+0000 7f336f0fd700 1 -- 172.21.15.52:0/2664727378 shutdown_connections 2024-02-19T22:08:36.736 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.733+0000 7f336f0fd700 1 --2- 172.21.15.52:0/2664727378 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f33680703d0 0x7f33680feab0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:36.736 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.733+0000 7f336f0fd700 1 -- 172.21.15.52:0/2664727378 >> 172.21.15.52:0/2664727378 conn(0x7f33680f6530 msgr2=0x7f33680f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:36.736 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.733+0000 7f336f0fd700 1 -- 172.21.15.52:0/2664727378 shutdown_connections 2024-02-19T22:08:36.736 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.734+0000 7f336f0fd700 1 -- 172.21.15.52:0/2664727378 wait complete. 2024-02-19T22:08:36.737 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.734+0000 7f336f0fd700 1 Processor -- start 2024-02-19T22:08:36.737 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.734+0000 7f336f0fd700 1 -- start start 2024-02-19T22:08:36.738 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.734+0000 7f336f0fd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f33680703d0 0x7f336810ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:36.738 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.734+0000 7f336f0fd700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f336810b060 con 0x7f33680703d0 2024-02-19T22:08:36.738 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.735+0000 7f336ce99700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f33680703d0 0x7f336810ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:36.738 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.735+0000 7f336ce99700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f33680703d0 0x7f336810ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:42426/0 (socket says 172.21.15.52:42426) 2024-02-19T22:08:36.738 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.735+0000 7f336ce99700 1 -- 172.21.15.52:0/3085997934 learned_addr learned my addr 172.21.15.52:0/3085997934 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:08:36.739 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.735+0000 7f336ce99700 1 -- 172.21.15.52:0/3085997934 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f335c00b200 con 0x7f33680703d0 2024-02-19T22:08:36.739 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.735+0000 7f336ce99700 1 --2- 172.21.15.52:0/3085997934 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f33680703d0 0x7f336810ab70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f335c006a20 tx=0x7f335c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:36.739 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.735+0000 7f33657fa700 1 -- 172.21.15.52:0/3085997934 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f335c01b570 con 0x7f33680703d0 2024-02-19T22:08:36.739 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.735+0000 7f33657fa700 1 -- 172.21.15.52:0/3085997934 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f335c01bb60 con 0x7f33680703d0 2024-02-19T22:08:36.739 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.735+0000 7f33657fa700 1 -- 172.21.15.52:0/3085997934 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f335c01aa20 con 0x7f33680703d0 2024-02-19T22:08:36.739 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.735+0000 7f336f0fd700 1 -- 172.21.15.52:0/3085997934 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f336810b260 con 0x7f33680703d0 2024-02-19T22:08:36.739 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.735+0000 7f336f0fd700 1 -- 172.21.15.52:0/3085997934 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f336810b6f0 con 0x7f33680703d0 2024-02-19T22:08:36.739 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.736+0000 7f33657fa700 1 -- 172.21.15.52:0/3085997934 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f335c00f070 con 0x7f33680703d0 2024-02-19T22:08:36.739 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.736+0000 7f33657fa700 1 -- 172.21.15.52:0/3085997934 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f335c025d00 con 0x7f33680703d0 2024-02-19T22:08:36.739 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.737+0000 7f336f0fd700 1 -- 172.21.15.52:0/3085997934 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3368045ad0 con 0x7f33680703d0 2024-02-19T22:08:36.740 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.739+0000 7f33657fa700 1 -- 172.21.15.52:0/3085997934 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f335c016070 con 0x7f33680703d0 2024-02-19T22:08:36.808 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.807+0000 7f336f0fd700 1 -- 172.21.15.52:0/3085997934 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f33680fff70 con 0x7f33680703d0 2024-02-19T22:08:36.808 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.807+0000 7f33657fa700 1 -- 172.21.15.52:0/3085997934 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f335c01a370 con 0x7f33680703d0 2024-02-19T22:08:36.809 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.808+0000 7f336f0fd700 1 -- 172.21.15.52:0/3085997934 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f33680703d0 msgr2=0x7f336810ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:36.809 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.808+0000 7f336f0fd700 1 --2- 172.21.15.52:0/3085997934 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f33680703d0 0x7f336810ab70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f335c006a20 tx=0x7f335c006ee0).stop 2024-02-19T22:08:36.809 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.808+0000 7f336f0fd700 1 -- 172.21.15.52:0/3085997934 shutdown_connections 2024-02-19T22:08:36.809 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.808+0000 7f336f0fd700 1 --2- 172.21.15.52:0/3085997934 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f33680703d0 0x7f336810ab70 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:36.809 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.808+0000 7f336f0fd700 1 -- 172.21.15.52:0/3085997934 >> 172.21.15.52:0/3085997934 conn(0x7f33680f6530 msgr2=0x7f33680f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:36.810 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.808+0000 7f336f0fd700 1 -- 172.21.15.52:0/3085997934 shutdown_connections 2024-02-19T22:08:36.810 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.808+0000 7f336f0fd700 1 -- 172.21.15.52:0/3085997934 wait complete. 2024-02-19T22:08:37.091 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:36 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[37748]: audit 2024-02-19T22:08:36.808662+0000 mon.smithi052 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.52:0/3085997934' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T22:08:37.433 INFO:teuthology.orchestra.run.smithi052.stderr:Restarting the monitor... 2024-02-19T22:08:37.433 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: systemctl restart ceph-56739a54-cf73-11ee-95bb-87774f69a715@mon.smithi052 2024-02-19T22:08:37.842 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:37 smithi052 systemd[1]: Stopping Ceph mon.smithi052 for 56739a54-cf73-11ee-95bb-87774f69a715... 2024-02-19T22:08:37.842 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:37 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[37748]: debug 2024-02-19T22:08:37.564+0000 7ffabd310700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi052 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-19T22:08:37.842 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:37 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[37748]: debug 2024-02-19T22:08:37.564+0000 7ffabd310700 -1 mon.smithi052@0(leader) e1 *** Got Signal Terminated *** 2024-02-19T22:08:38.532 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:38 smithi052 podman[38072]: ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052 2024-02-19T22:08:38.841 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:38 smithi052 systemd[1]: ceph-56739a54-cf73-11ee-95bb-87774f69a715@mon.smithi052.service: Succeeded. 2024-02-19T22:08:38.842 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:38 smithi052 systemd[1]: Stopped Ceph mon.smithi052 for 56739a54-cf73-11ee-95bb-87774f69a715. 2024-02-19T22:08:38.842 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:38 smithi052 systemd[1]: Starting Ceph mon.smithi052 for 56739a54-cf73-11ee-95bb-87774f69a715... 2024-02-19T22:08:39.165 INFO:teuthology.orchestra.run.smithi052.stderr:Setting mon public_network to 172.21.0.0/20 2024-02-19T22:08:39.166 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi052 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphvym4zi3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp6ou7w4ul:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2024-02-19T22:08:39.343 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.053+0000 7f69cf21d700 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-19T22:08:39.343 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.053+0000 7f69cf21d700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2024-02-19T22:08:39.344 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.053+0000 7f69cf21d700 0 pidfile_write: ignore empty --pid-file 2024-02-19T22:08:39.344 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.057+0000 7f69cf21d700 0 load: jerasure load: lrc load: isa 2024-02-19T22:08:39.344 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: RocksDB version: 6.8.1 2024-02-19T22:08:39.344 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: 2024-02-19T22:08:39.344 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-19T22:08:39.344 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Compile date May 13 2021 2024-02-19T22:08:39.344 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: DB SUMMARY 2024-02-19T22:08:39.344 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: 2024-02-19T22:08:39.344 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: CURRENT file: CURRENT 2024-02-19T22:08:39.344 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: 2024-02-19T22:08:39.344 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: IDENTITY file: IDENTITY 2024-02-19T22:08:39.344 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: 2024-02-19T22:08:39.345 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2024-02-19T22:08:39.345 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: 2024-02-19T22:08:39.345 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi052/store.db dir, Total Num: 1, files: 000004.sst 2024-02-19T22:08:39.345 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: 2024-02-19T22:08:39.345 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi052/store.db: 000006.log size: 98702 ; 2024-02-19T22:08:39.345 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: 2024-02-19T22:08:39.345 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.error_if_exists: 0 2024-02-19T22:08:39.345 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.create_if_missing: 0 2024-02-19T22:08:39.345 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.paranoid_checks: 1 2024-02-19T22:08:39.345 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.env: 0x55ee429081c0 2024-02-19T22:08:39.345 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.fs: Posix File System 2024-02-19T22:08:39.346 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.info_log: 0x55ee44e67960 2024-02-19T22:08:39.346 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-19T22:08:39.346 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.statistics: (nil) 2024-02-19T22:08:39.346 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.use_fsync: 0 2024-02-19T22:08:39.346 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.max_log_file_size: 0 2024-02-19T22:08:39.346 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-19T22:08:39.346 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-19T22:08:39.346 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-19T22:08:39.346 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-19T22:08:39.346 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.allow_fallocate: 1 2024-02-19T22:08:39.346 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-19T22:08:39.347 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-19T22:08:39.347 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.use_direct_reads: 0 2024-02-19T22:08:39.347 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-19T22:08:39.347 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.create_missing_column_families: 0 2024-02-19T22:08:39.347 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.db_log_dir: 2024-02-19T22:08:39.347 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi052/store.db 2024-02-19T22:08:39.347 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-19T22:08:39.347 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.max_subcompactions: 1 2024-02-19T22:08:39.347 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.max_background_flushes: -1 2024-02-19T22:08:39.347 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-19T22:08:39.348 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-19T22:08:39.348 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-19T22:08:39.348 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-19T22:08:39.348 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-19T22:08:39.348 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.advise_random_on_open: 1 2024-02-19T22:08:39.348 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-19T22:08:39.348 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.write_buffer_manager: 0x55ee45adc030 2024-02-19T22:08:39.348 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-19T22:08:39.348 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-19T22:08:39.348 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-19T22:08:39.348 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-19T22:08:39.348 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.rate_limiter: (nil) 2024-02-19T22:08:39.348 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-19T22:08:39.348 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-19T22:08:39.348 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-19T22:08:39.349 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-19T22:08:39.349 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.unordered_write: 0 2024-02-19T22:08:39.349 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-19T22:08:39.349 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-19T22:08:39.349 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-19T22:08:39.349 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-19T22:08:39.349 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.row_cache: None 2024-02-19T22:08:39.349 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.wal_filter: None 2024-02-19T22:08:39.349 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-19T22:08:39.349 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-19T22:08:39.349 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.preserve_deletes: 0 2024-02-19T22:08:39.349 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.two_write_queues: 0 2024-02-19T22:08:39.349 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.manual_wal_flush: 0 2024-02-19T22:08:39.349 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.atomic_flush: 0 2024-02-19T22:08:39.349 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-19T22:08:39.349 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-19T22:08:39.350 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.058+0000 7f69cf21d700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-19T22:08:39.350 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.log_readahead_size: 0 2024-02-19T22:08:39.350 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-19T22:08:39.350 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.max_background_jobs: 2 2024-02-19T22:08:39.350 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.max_background_compactions: -1 2024-02-19T22:08:39.350 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-19T22:08:39.350 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-19T22:08:39.350 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-19T22:08:39.350 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.max_total_wal_size: 0 2024-02-19T22:08:39.350 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-19T22:08:39.350 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-19T22:08:39.350 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-19T22:08:39.350 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-19T22:08:39.350 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.max_open_files: -1 2024-02-19T22:08:39.350 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.bytes_per_sync: 0 2024-02-19T22:08:39.351 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-19T22:08:39.351 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-19T22:08:39.351 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-19T22:08:39.351 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Compression algorithms supported: 2024-02-19T22:08:39.351 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-19T22:08:39.351 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: kZSTD supported: 0 2024-02-19T22:08:39.351 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: kXpressCompression supported: 0 2024-02-19T22:08:39.351 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-19T22:08:39.351 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: kLZ4Compression supported: 1 2024-02-19T22:08:39.351 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: kBZip2Compression supported: 0 2024-02-19T22:08:39.351 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: kZlibCompression supported: 1 2024-02-19T22:08:39.351 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: kSnappyCompression supported: 1 2024-02-19T22:08:39.351 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-19T22:08:39.351 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi052/store.db/MANIFEST-000005 2024-02-19T22:08:39.351 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: 2024-02-19T22:08:39.351 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-19T22:08:39.352 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: 2024-02-19T22:08:39.352 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-19T22:08:39.352 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.merge_operator: 2024-02-19T22:08:39.352 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.compaction_filter: None 2024-02-19T22:08:39.352 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.compaction_filter_factory: None 2024-02-19T22:08:39.352 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-19T22:08:39.352 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-19T22:08:39.352 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ee44d7fcb0) 2024-02-19T22:08:39.352 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cache_index_and_filter_blocks: 1 2024-02-19T22:08:39.352 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-19T22:08:39.352 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-19T22:08:39.353 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: pin_top_level_index_and_filter: 1 2024-02-19T22:08:39.353 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: index_type: 0 2024-02-19T22:08:39.353 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: data_block_index_type: 0 2024-02-19T22:08:39.353 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: index_shortening: 1 2024-02-19T22:08:39.353 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: data_block_hash_table_util_ratio: 0.750000 2024-02-19T22:08:39.353 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: hash_index_allow_collision: 1 2024-02-19T22:08:39.353 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: checksum: 1 2024-02-19T22:08:39.353 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: no_block_cache: 0 2024-02-19T22:08:39.353 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: block_cache: 0x55ee44db4f10 2024-02-19T22:08:39.353 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: block_cache_name: BinnedLRUCache 2024-02-19T22:08:39.353 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: block_cache_options: 2024-02-19T22:08:39.353 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: capacity : 536870912 2024-02-19T22:08:39.353 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: num_shard_bits : 4 2024-02-19T22:08:39.353 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: strict_capacity_limit : 0 2024-02-19T22:08:39.353 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: high_pri_pool_ratio: 0.000 2024-02-19T22:08:39.354 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: block_cache_compressed: (nil) 2024-02-19T22:08:39.354 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: persistent_cache: (nil) 2024-02-19T22:08:39.354 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: block_size: 4096 2024-02-19T22:08:39.354 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: block_size_deviation: 10 2024-02-19T22:08:39.354 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: block_restart_interval: 16 2024-02-19T22:08:39.354 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: index_block_restart_interval: 1 2024-02-19T22:08:39.354 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: metadata_block_size: 4096 2024-02-19T22:08:39.354 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: partition_filters: 0 2024-02-19T22:08:39.354 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: use_delta_encoding: 1 2024-02-19T22:08:39.354 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-19T22:08:39.354 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: whole_key_filtering: 1 2024-02-19T22:08:39.354 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: verify_compression: 0 2024-02-19T22:08:39.354 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: read_amp_bytes_per_bit: 0 2024-02-19T22:08:39.354 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: format_version: 2 2024-02-19T22:08:39.355 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: enable_index_compression: 1 2024-02-19T22:08:39.355 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: block_align: 0 2024-02-19T22:08:39.355 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: 2024-02-19T22:08:39.355 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-19T22:08:39.355 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-19T22:08:39.355 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.compression: NoCompression 2024-02-19T22:08:39.355 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-19T22:08:39.355 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-19T22:08:39.355 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-19T22:08:39.355 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.num_levels: 7 2024-02-19T22:08:39.355 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-19T22:08:39.355 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-19T22:08:39.355 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-19T22:08:39.356 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-19T22:08:39.356 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-19T22:08:39.356 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-19T22:08:39.356 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-19T22:08:39.356 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-19T22:08:39.356 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-19T22:08:39.356 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-19T22:08:39.356 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.compression_opts.level: 32767 2024-02-19T22:08:39.356 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-19T22:08:39.356 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-19T22:08:39.356 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-19T22:08:39.356 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.compression_opts.enabled: false 2024-02-19T22:08:39.356 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-19T22:08:39.357 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-19T22:08:39.357 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-19T22:08:39.357 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-19T22:08:39.357 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-19T22:08:39.357 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-19T22:08:39.357 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-19T22:08:39.357 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-19T22:08:39.357 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-19T22:08:39.357 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-19T22:08:39.357 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-19T22:08:39.357 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-19T22:08:39.357 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-19T22:08:39.357 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-19T22:08:39.358 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-19T22:08:39.358 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-19T22:08:39.358 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-19T22:08:39.358 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.arena_block_size: 4194304 2024-02-19T22:08:39.358 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-19T22:08:39.358 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-19T22:08:39.358 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-19T22:08:39.358 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-19T22:08:39.358 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-19T22:08:39.358 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-19T22:08:39.358 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-19T22:08:39.358 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-19T22:08:39.359 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-19T22:08:39.359 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-19T22:08:39.359 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-19T22:08:39.359 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-19T22:08:39.359 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-19T22:08:39.359 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-19T22:08:39.359 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.table_properties_collectors: 2024-02-19T22:08:39.359 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.inplace_update_support: 0 2024-02-19T22:08:39.359 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-19T22:08:39.359 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-19T22:08:39.359 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-19T22:08:39.359 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-19T22:08:39.359 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.bloom_locality: 0 2024-02-19T22:08:39.359 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.max_successive_merges: 0 2024-02-19T22:08:39.359 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-19T22:08:39.360 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-19T22:08:39.360 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.force_consistency_checks: 0 2024-02-19T22:08:39.360 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-19T22:08:39.360 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.ttl: 2592000 2024-02-19T22:08:39.360 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-19T22:08:39.360 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.059+0000 7f69cf21d700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi052/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-19T22:08:39.360 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: 2024-02-19T22:08:39.360 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.060+0000 7f69cf21d700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2024-02-19T22:08:39.360 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: 2024-02-19T22:08:39.360 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.060+0000 7f69cf21d700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708380519061348, "job": 1, "event": "recovery_started", "log_files": [6]} 2024-02-19T22:08:39.360 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.060+0000 7f69cf21d700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2024-02-19T22:08:39.360 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.060+0000 7f69cf21d700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-19T22:08:39.360 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.061+0000 7f69cf21d700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708380519062427, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 94276, "table_properties": {"data_size": 92364, "index_size": 389, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12563, "raw_average_key_size": 49, "raw_value_size": 85395, "raw_average_value_size": 336, "num_data_blocks": 12, "num_entries": 254, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1708380519, "oldest_key_time": 3, "file_creation_time": 0}} 2024-02-19T22:08:39.360 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.061+0000 7f69cf21d700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2024-02-19T22:08:39.361 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: 2024-02-19T22:08:39.361 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.061+0000 7f69cf21d700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708380519062909, "job": 1, "event": "recovery_finished"} 2024-02-19T22:08:39.361 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.062+0000 7f69cf21d700 4 rocksdb: DB pointer 0x55ee44e79800 2024-02-19T22:08:39.361 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.062+0000 7f69b6035700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-19T22:08:39.361 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.062+0000 7f69b6035700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-19T22:08:39.361 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: ** DB Stats ** 2024-02-19T22:08:39.361 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T22:08:39.361 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-19T22:08:39.361 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-19T22:08:39.361 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T22:08:39.362 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-19T22:08:39.362 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-19T22:08:39.362 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T22:08:39.362 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: 2024-02-19T22:08:39.363 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: ** Compaction Stats [default] ** 2024-02-19T22:08:39.363 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T22:08:39.363 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T22:08:39.363 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: L0 2/0 93.76 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 123.2 0.00 0.00 1 0.001 0 0 2024-02-19T22:08:39.363 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: Sum 2/0 93.76 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 123.2 0.00 0.00 1 0.001 0 0 2024-02-19T22:08:39.363 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 123.2 0.00 0.00 1 0.001 0 0 2024-02-19T22:08:39.363 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: 2024-02-19T22:08:39.363 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: ** Compaction Stats [default] ** 2024-02-19T22:08:39.363 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T22:08:39.363 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T22:08:39.363 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 123.2 0.00 0.00 1 0.001 0 0 2024-02-19T22:08:39.363 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T22:08:39.364 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-19T22:08:39.364 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-19T22:08:39.364 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: AddFile(Total Files): cumulative 0, interval 0 2024-02-19T22:08:39.364 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-19T22:08:39.364 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: AddFile(Keys): cumulative 0, interval 0 2024-02-19T22:08:39.364 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: Cumulative compaction: 0.00 GB write, 26.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T22:08:39.365 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: Interval compaction: 0.00 GB write, 26.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T22:08:39.365 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-19T22:08:39.365 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: 2024-02-19T22:08:39.365 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: ** File Read Latency Histogram By Level [default] ** 2024-02-19T22:08:39.365 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: 2024-02-19T22:08:39.365 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: ** Compaction Stats [default] ** 2024-02-19T22:08:39.365 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T22:08:39.365 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T22:08:39.366 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: L0 2/0 93.76 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 123.2 0.00 0.00 1 0.001 0 0 2024-02-19T22:08:39.366 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: Sum 2/0 93.76 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 123.2 0.00 0.00 1 0.001 0 0 2024-02-19T22:08:39.366 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-19T22:08:39.366 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: 2024-02-19T22:08:39.366 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: ** Compaction Stats [default] ** 2024-02-19T22:08:39.366 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T22:08:39.366 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T22:08:39.366 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 123.2 0.00 0.00 1 0.001 0 0 2024-02-19T22:08:39.366 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T22:08:39.366 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-19T22:08:39.366 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-19T22:08:39.366 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: AddFile(Total Files): cumulative 0, interval 0 2024-02-19T22:08:39.366 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-19T22:08:39.366 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: AddFile(Keys): cumulative 0, interval 0 2024-02-19T22:08:39.366 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: Cumulative compaction: 0.00 GB write, 26.30 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T22:08:39.367 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T22:08:39.367 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-19T22:08:39.367 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: 2024-02-19T22:08:39.367 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: ** File Read Latency Histogram By Level [default] ** 2024-02-19T22:08:39.367 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: 2024-02-19T22:08:39.367 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.063+0000 7f69cf21d700 0 starting mon.smithi052 rank 0 at public addrs [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] at bind addrs [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi052 fsid 56739a54-cf73-11ee-95bb-87774f69a715 2024-02-19T22:08:39.367 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.064+0000 7f69cf21d700 1 mon.smithi052@-1(???) e1 preinit fsid 56739a54-cf73-11ee-95bb-87774f69a715 2024-02-19T22:08:39.367 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.064+0000 7f69cf21d700 0 mon.smithi052@-1(???).mds e1 new map 2024-02-19T22:08:39.367 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.064+0000 7f69cf21d700 0 mon.smithi052@-1(???).mds e1 print_map 2024-02-19T22:08:39.367 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: e1 2024-02-19T22:08:39.367 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T22:08:39.367 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T22:08:39.367 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: legacy client fscid: -1 2024-02-19T22:08:39.368 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: 2024-02-19T22:08:39.368 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: No filesystems configured 2024-02-19T22:08:39.368 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: 2024-02-19T22:08:39.368 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.064+0000 7f69cf21d700 0 mon.smithi052@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-19T22:08:39.368 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.064+0000 7f69cf21d700 0 mon.smithi052@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-19T22:08:39.368 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.064+0000 7f69cf21d700 0 mon.smithi052@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-19T22:08:39.368 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.064+0000 7f69cf21d700 0 mon.smithi052@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-19T22:08:39.368 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:08:36.808662+0000 mon.smithi052 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.52:0/3085997934' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T22:08:39.368 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:08:39.064+0000 7f69cf21d700 1 mon.smithi052@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-19T22:08:39.369 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:08:39.082062+0000 mon.smithi052 (mon.0) 1 : cluster [INF] mon.smithi052 is new leader, mons smithi052 in quorum (ranks 0) 2024-02-19T22:08:39.369 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:08:39.082112+0000 mon.smithi052 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi052=[v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0]} 2024-02-19T22:08:39.369 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:08:39.082156+0000 mon.smithi052 (mon.0) 3 : cluster [DBG] fsmap 2024-02-19T22:08:39.369 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:08:39.082181+0000 mon.smithi052 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-02-19T22:08:39.369 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:08:39.082545+0000 mon.smithi052 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2024-02-19T22:08:39.369 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 bash[38142]: e98b0f1dc2f8656ddbdf1f3fe51aa00cc9161b7a69dbb8a6ff1565abd5963248 2024-02-19T22:08:39.369 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:39 smithi052 systemd[1]: Started Ceph mon.smithi052 for 56739a54-cf73-11ee-95bb-87774f69a715. 2024-02-19T22:08:39.791 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.785+0000 7f9f4ea37700 1 Processor -- start 2024-02-19T22:08:39.792 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.785+0000 7f9f4ea37700 1 -- start start 2024-02-19T22:08:39.792 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.785+0000 7f9f4ea37700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f48072560 0x7f9f48070b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:39.792 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.785+0000 7f9f4ea37700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f48072980 con 0x7f9f48072560 2024-02-19T22:08:39.792 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.786+0000 7f9f47fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f48072560 0x7f9f48070b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:39.792 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.786+0000 7f9f47fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f48072560 0x7f9f48070b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:42436/0 (socket says 172.21.15.52:42436) 2024-02-19T22:08:39.792 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.786+0000 7f9f47fff700 1 -- 172.21.15.52:0/1781938421 learned_addr learned my addr 172.21.15.52:0/1781938421 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:08:39.792 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.786+0000 7f9f47fff700 1 -- 172.21.15.52:0/1781938421 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f48070ff0 con 0x7f9f48072560 2024-02-19T22:08:39.792 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.786+0000 7f9f47fff700 1 --2- 172.21.15.52:0/1781938421 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f48072560 0x7f9f48070b00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f9f3800b5f0 tx=0x7f9f38006ee0).ready entity=mon.0 client_cookie=576340930cbd43ba server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:39.792 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.787+0000 7f9f46ffd700 1 -- 172.21.15.52:0/1781938421 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f3800d600 con 0x7f9f48072560 2024-02-19T22:08:39.792 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.787+0000 7f9f46ffd700 1 -- 172.21.15.52:0/1781938421 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f9f3800dbf0 con 0x7f9f48072560 2024-02-19T22:08:39.792 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.787+0000 7f9f46ffd700 1 -- 172.21.15.52:0/1781938421 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f38011cb0 con 0x7f9f48072560 2024-02-19T22:08:39.792 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.787+0000 7f9f4ea37700 1 -- 172.21.15.52:0/1781938421 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f48072560 msgr2=0x7f9f48070b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:39.792 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.787+0000 7f9f4ea37700 1 --2- 172.21.15.52:0/1781938421 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f48072560 0x7f9f48070b00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f9f3800b5f0 tx=0x7f9f38006ee0).stop 2024-02-19T22:08:39.792 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.787+0000 7f9f4ea37700 1 -- 172.21.15.52:0/1781938421 shutdown_connections 2024-02-19T22:08:39.793 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.787+0000 7f9f4ea37700 1 --2- 172.21.15.52:0/1781938421 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f48072560 0x7f9f48070b00 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:39.793 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.787+0000 7f9f4ea37700 1 -- 172.21.15.52:0/1781938421 >> 172.21.15.52:0/1781938421 conn(0x7f9f480f76a0 msgr2=0x7f9f480f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:39.793 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.787+0000 7f9f4ea37700 1 -- 172.21.15.52:0/1781938421 shutdown_connections 2024-02-19T22:08:39.793 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.787+0000 7f9f4ea37700 1 -- 172.21.15.52:0/1781938421 wait complete. 2024-02-19T22:08:39.793 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.788+0000 7f9f4ea37700 1 Processor -- start 2024-02-19T22:08:39.793 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.788+0000 7f9f4ea37700 1 -- start start 2024-02-19T22:08:39.793 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.788+0000 7f9f4ea37700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f48072560 0x7f9f4810ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:39.793 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.788+0000 7f9f4ea37700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4810d350 con 0x7f9f48072560 2024-02-19T22:08:39.793 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.789+0000 7f9f47fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f48072560 0x7f9f4810ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:39.793 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.789+0000 7f9f47fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f48072560 0x7f9f4810ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:42442/0 (socket says 172.21.15.52:42442) 2024-02-19T22:08:39.793 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.789+0000 7f9f47fff700 1 -- 172.21.15.52:0/328434553 learned_addr learned my addr 172.21.15.52:0/328434553 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:08:39.793 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.789+0000 7f9f47fff700 1 -- 172.21.15.52:0/328434553 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f3800b200 con 0x7f9f48072560 2024-02-19T22:08:39.793 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.789+0000 7f9f47fff700 1 --2- 172.21.15.52:0/328434553 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f48072560 0x7f9f4810ce60 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f9f3800df90 tx=0x7f9f38006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:39.793 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.789+0000 7f9f44ff9700 1 -- 172.21.15.52:0/328434553 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f3801b410 con 0x7f9f48072560 2024-02-19T22:08:39.793 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.789+0000 7f9f44ff9700 1 -- 172.21.15.52:0/328434553 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f9f3801ba00 con 0x7f9f48072560 2024-02-19T22:08:39.794 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.789+0000 7f9f4ea37700 1 -- 172.21.15.52:0/328434553 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f4810d550 con 0x7f9f48072560 2024-02-19T22:08:39.794 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.789+0000 7f9f44ff9700 1 -- 172.21.15.52:0/328434553 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f3801aa30 con 0x7f9f48072560 2024-02-19T22:08:39.794 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.789+0000 7f9f4ea37700 1 -- 172.21.15.52:0/328434553 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f4810e160 con 0x7f9f48072560 2024-02-19T22:08:39.794 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.790+0000 7f9f44ff9700 1 -- 172.21.15.52:0/328434553 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f9f3800f070 con 0x7f9f48072560 2024-02-19T22:08:39.794 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.790+0000 7f9f44ff9700 1 -- 172.21.15.52:0/328434553 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f9f38025c70 con 0x7f9f48072560 2024-02-19T22:08:39.794 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.790+0000 7f9f4ea37700 1 -- 172.21.15.52:0/328434553 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f48059100 con 0x7f9f48072560 2024-02-19T22:08:39.794 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.793+0000 7f9f44ff9700 1 -- 172.21.15.52:0/328434553 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f9f38016070 con 0x7f9f48072560 2024-02-19T22:08:39.860 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.859+0000 7f9f4ea37700 1 -- 172.21.15.52:0/328434553 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f9f4810ef90 con 0x7f9f48072560 2024-02-19T22:08:39.862 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.861+0000 7f9f44ff9700 1 -- 172.21.15.52:0/328434553 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f9f38025e30 con 0x7f9f48072560 2024-02-19T22:08:39.863 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.862+0000 7f9f4ea37700 1 -- 172.21.15.52:0/328434553 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f48072560 msgr2=0x7f9f4810ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:39.863 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.862+0000 7f9f4ea37700 1 --2- 172.21.15.52:0/328434553 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f48072560 0x7f9f4810ce60 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f9f3800df90 tx=0x7f9f38006ee0).stop 2024-02-19T22:08:39.863 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.862+0000 7f9f4ea37700 1 -- 172.21.15.52:0/328434553 shutdown_connections 2024-02-19T22:08:39.863 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.862+0000 7f9f4ea37700 1 --2- 172.21.15.52:0/328434553 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f48072560 0x7f9f4810ce60 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:39.863 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.862+0000 7f9f4ea37700 1 -- 172.21.15.52:0/328434553 >> 172.21.15.52:0/328434553 conn(0x7f9f480f76a0 msgr2=0x7f9f480f90c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:39.863 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.862+0000 7f9f4ea37700 1 -- 172.21.15.52:0/328434553 shutdown_connections 2024-02-19T22:08:39.863 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.862+0000 7f9f4ea37700 1 -- 172.21.15.52:0/328434553 wait complete. 2024-02-19T22:08:40.480 INFO:teuthology.orchestra.run.smithi052.stderr:Wrote config to /etc/ceph/ceph.conf 2024-02-19T22:08:40.480 INFO:teuthology.orchestra.run.smithi052.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-19T22:08:40.480 INFO:teuthology.orchestra.run.smithi052.stderr:Creating mgr... 2024-02-19T22:08:40.480 INFO:teuthology.orchestra.run.smithi052.stderr:Verifying port 9283 ... 2024-02-19T22:08:40.483 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: systemctl daemon-reload 2024-02-19T22:08:40.588 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: systemctl stop ceph-56739a54-cf73-11ee-95bb-87774f69a715@mgr.smithi052.ghqrtk 2024-02-19T22:08:40.623 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: systemctl reset-failed ceph-56739a54-cf73-11ee-95bb-87774f69a715@mgr.smithi052.ghqrtk 2024-02-19T22:08:40.629 INFO:teuthology.orchestra.run.smithi052.stderr:systemctl: Failed to reset failed state of unit ceph-56739a54-cf73-11ee-95bb-87774f69a715@mgr.smithi052.ghqrtk.service: Unit ceph-56739a54-cf73-11ee-95bb-87774f69a715@mgr.smithi052.ghqrtk.service not loaded. 2024-02-19T22:08:40.629 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: systemctl enable ceph-56739a54-cf73-11ee-95bb-87774f69a715@mgr.smithi052.ghqrtk 2024-02-19T22:08:40.635 INFO:teuthology.orchestra.run.smithi052.stderr:systemctl: Created symlink /etc/systemd/system/ceph-56739a54-cf73-11ee-95bb-87774f69a715.target.wants/ceph-56739a54-cf73-11ee-95bb-87774f69a715@mgr.smithi052.ghqrtk.service → /etc/systemd/system/ceph-56739a54-cf73-11ee-95bb-87774f69a715@.service. 2024-02-19T22:08:40.725 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: systemctl start ceph-56739a54-cf73-11ee-95bb-87774f69a715@mgr.smithi052.ghqrtk 2024-02-19T22:08:41.342 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:40 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:08:39.862421+0000 mon.smithi052 (mon.0) 6 : audit [INF] from='client.? 172.21.15.52:0/328434553' entity='client.admin' 2024-02-19T22:08:41.369 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-19T22:08:41.379 INFO:teuthology.orchestra.run.smithi052.stderr:systemctl: disabled 2024-02-19T22:08:41.380 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: systemctl is-active firewalld.service 2024-02-19T22:08:41.388 INFO:teuthology.orchestra.run.smithi052.stderr:systemctl: inactive 2024-02-19T22:08:41.389 INFO:teuthology.orchestra.run.smithi052.stderr:firewalld.service is not enabled 2024-02-19T22:08:41.389 INFO:teuthology.orchestra.run.smithi052.stderr:Not possible to enable service . firewalld.service is not available 2024-02-19T22:08:41.389 INFO:teuthology.orchestra.run.smithi052.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-02-19T22:08:41.389 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-19T22:08:41.398 INFO:teuthology.orchestra.run.smithi052.stderr:systemctl: disabled 2024-02-19T22:08:41.398 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: systemctl is-active firewalld.service 2024-02-19T22:08:41.406 INFO:teuthology.orchestra.run.smithi052.stderr:systemctl: inactive 2024-02-19T22:08:41.408 INFO:teuthology.orchestra.run.smithi052.stderr:firewalld.service is not enabled 2024-02-19T22:08:41.408 INFO:teuthology.orchestra.run.smithi052.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-02-19T22:08:41.408 INFO:teuthology.orchestra.run.smithi052.stderr:Waiting for mgr to start... 2024-02-19T22:08:41.408 INFO:teuthology.orchestra.run.smithi052.stderr:Waiting for mgr... 2024-02-19T22:08:41.408 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi052 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphvym4zi3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp6ou7w4ul:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-02-19T22:08:41.948 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.946+0000 7fccbec0c700 1 Processor -- start 2024-02-19T22:08:41.949 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.946+0000 7fccbec0c700 1 -- start start 2024-02-19T22:08:41.949 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.947+0000 7fccbec0c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fccb80703b0 0x7fccb8100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:41.949 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.947+0000 7fccbec0c700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb80fc8c0 con 0x7fccb80703b0 2024-02-19T22:08:41.949 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.947+0000 7fccbc9a8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fccb80703b0 0x7fccb8100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:41.949 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.947+0000 7fccbc9a8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fccb80703b0 0x7fccb8100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:42472/0 (socket says 172.21.15.52:42472) 2024-02-19T22:08:41.949 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.947+0000 7fccbc9a8700 1 -- 172.21.15.52:0/1068469362 learned_addr learned my addr 172.21.15.52:0/1068469362 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:08:41.950 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.947+0000 7fccbc9a8700 1 -- 172.21.15.52:0/1068469362 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb81013c0 con 0x7fccb80703b0 2024-02-19T22:08:41.950 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.948+0000 7fccbc9a8700 1 --2- 172.21.15.52:0/1068469362 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fccb80703b0 0x7fccb8100ed0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fccac00b5f0 tx=0x7fccac006ee0).ready entity=mon.0 client_cookie=43043e3e0b150343 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:41.950 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.948+0000 7fccb77fe700 1 -- 172.21.15.52:0/1068469362 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fccac00d600 con 0x7fccb80703b0 2024-02-19T22:08:41.950 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.948+0000 7fccb77fe700 1 -- 172.21.15.52:0/1068469362 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fccac00dbf0 con 0x7fccb80703b0 2024-02-19T22:08:41.950 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.948+0000 7fccb77fe700 1 -- 172.21.15.52:0/1068469362 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fccac011cb0 con 0x7fccb80703b0 2024-02-19T22:08:41.950 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.948+0000 7fccbec0c700 1 -- 172.21.15.52:0/1068469362 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fccb80703b0 msgr2=0x7fccb8100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:41.950 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.948+0000 7fccbec0c700 1 --2- 172.21.15.52:0/1068469362 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fccb80703b0 0x7fccb8100ed0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fccac00b5f0 tx=0x7fccac006ee0).stop 2024-02-19T22:08:41.951 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.948+0000 7fccbec0c700 1 -- 172.21.15.52:0/1068469362 shutdown_connections 2024-02-19T22:08:41.951 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.948+0000 7fccbec0c700 1 --2- 172.21.15.52:0/1068469362 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fccb80703b0 0x7fccb8100ed0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:41.951 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.948+0000 7fccbec0c700 1 -- 172.21.15.52:0/1068469362 >> 172.21.15.52:0/1068469362 conn(0x7fccb80f6530 msgr2=0x7fccb80f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:41.951 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.949+0000 7fccbec0c700 1 -- 172.21.15.52:0/1068469362 shutdown_connections 2024-02-19T22:08:41.951 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.949+0000 7fccbec0c700 1 -- 172.21.15.52:0/1068469362 wait complete. 2024-02-19T22:08:41.951 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.949+0000 7fccbec0c700 1 Processor -- start 2024-02-19T22:08:41.951 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.949+0000 7fccbec0c700 1 -- start start 2024-02-19T22:08:41.951 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.949+0000 7fccbec0c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fccb80703b0 0x7fccb810cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:41.953 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.949+0000 7fccbec0c700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb810d2c0 con 0x7fccb80703b0 2024-02-19T22:08:41.953 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.950+0000 7fccbc9a8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fccb80703b0 0x7fccb810cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:41.953 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.950+0000 7fccbc9a8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fccb80703b0 0x7fccb810cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:42478/0 (socket says 172.21.15.52:42478) 2024-02-19T22:08:41.953 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.950+0000 7fccbc9a8700 1 -- 172.21.15.52:0/2090528890 learned_addr learned my addr 172.21.15.52:0/2090528890 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:08:41.953 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.950+0000 7fccbc9a8700 1 -- 172.21.15.52:0/2090528890 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccac00b200 con 0x7fccb80703b0 2024-02-19T22:08:41.953 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.950+0000 7fccbc9a8700 1 --2- 172.21.15.52:0/2090528890 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fccb80703b0 0x7fccb810cdd0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fccac00df90 tx=0x7fccac006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:41.953 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.950+0000 7fccb57fa700 1 -- 172.21.15.52:0/2090528890 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fccac01b410 con 0x7fccb80703b0 2024-02-19T22:08:41.953 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.950+0000 7fccb57fa700 1 -- 172.21.15.52:0/2090528890 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fccac01ba00 con 0x7fccb80703b0 2024-02-19T22:08:41.953 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.950+0000 7fccbec0c700 1 -- 172.21.15.52:0/2090528890 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccb810d4c0 con 0x7fccb80703b0 2024-02-19T22:08:41.953 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.950+0000 7fccb57fa700 1 -- 172.21.15.52:0/2090528890 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fccac01aa30 con 0x7fccb80703b0 2024-02-19T22:08:41.953 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.950+0000 7fccbec0c700 1 -- 172.21.15.52:0/2090528890 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccb810e100 con 0x7fccb80703b0 2024-02-19T22:08:41.953 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.951+0000 7fccb57fa700 1 -- 172.21.15.52:0/2090528890 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fccac00f070 con 0x7fccb80703b0 2024-02-19T22:08:41.954 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.951+0000 7fccb57fa700 1 -- 172.21.15.52:0/2090528890 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fccac025cd0 con 0x7fccb80703b0 2024-02-19T22:08:41.954 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.951+0000 7fccbec0c700 1 -- 172.21.15.52:0/2090528890 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccb810d950 con 0x7fccb80703b0 2024-02-19T22:08:41.954 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.954+0000 7fccb57fa700 1 -- 172.21.15.52:0/2090528890 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fccac016070 con 0x7fccb80703b0 2024-02-19T22:08:42.025 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:42.023+0000 7fccbec0c700 1 -- 172.21.15.52:0/2090528890 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fccb81020e0 con 0x7fccb80703b0 2024-02-19T22:08:42.025 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:42.023+0000 7fccb57fa700 1 -- 172.21.15.52:0/2090528890 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7fccac025460 con 0x7fccb80703b0 2024-02-19T22:08:42.025 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:42.025 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: { 2024-02-19T22:08:42.025 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "fsid": "56739a54-cf73-11ee-95bb-87774f69a715", 2024-02-19T22:08:42.025 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "health": { 2024-02-19T22:08:42.025 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-19T22:08:42.025 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "checks": {}, 2024-02-19T22:08:42.026 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "mutes": [] 2024-02-19T22:08:42.027 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: }, 2024-02-19T22:08:42.027 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-19T22:08:42.027 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "quorum": [ 2024-02-19T22:08:42.027 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 0 2024-02-19T22:08:42.027 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: ], 2024-02-19T22:08:42.027 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-19T22:08:42.027 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "smithi052" 2024-02-19T22:08:42.027 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: ], 2024-02-19T22:08:42.027 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-02-19T22:08:42.027 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "monmap": { 2024-02-19T22:08:42.031 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:08:42.032 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-19T22:08:42.032 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-19T22:08:42.032 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: }, 2024-02-19T22:08:42.032 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "osdmap": { 2024-02-19T22:08:42.032 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:08:42.032 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-19T22:08:42.032 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-19T22:08:42.032 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-19T22:08:42.032 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-19T22:08:42.032 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-19T22:08:42.032 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-19T22:08:42.032 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: }, 2024-02-19T22:08:42.032 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "pgmap": { 2024-02-19T22:08:42.033 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-19T22:08:42.033 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-19T22:08:42.033 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-19T22:08:42.033 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-19T22:08:42.033 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-19T22:08:42.033 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-19T22:08:42.033 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-19T22:08:42.033 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-19T22:08:42.033 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: }, 2024-02-19T22:08:42.033 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "fsmap": { 2024-02-19T22:08:42.033 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:08:42.033 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-19T22:08:42.033 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-19T22:08:42.033 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: }, 2024-02-19T22:08:42.033 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-19T22:08:42.034 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "available": false, 2024-02-19T22:08:42.034 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-19T22:08:42.034 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "modules": [ 2024-02-19T22:08:42.034 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "iostat", 2024-02-19T22:08:42.034 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "restful" 2024-02-19T22:08:42.034 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: ], 2024-02-19T22:08:42.034 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "services": {} 2024-02-19T22:08:42.034 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: }, 2024-02-19T22:08:42.034 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "servicemap": { 2024-02-19T22:08:42.034 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:08:42.034 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "modified": "2024-02-19T22:08:33.546528+0000", 2024-02-19T22:08:42.034 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "services": {} 2024-02-19T22:08:42.034 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: }, 2024-02-19T22:08:42.034 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-19T22:08:42.034 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: } 2024-02-19T22:08:42.035 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:42.024+0000 7fccbec0c700 1 -- 172.21.15.52:0/2090528890 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fccb80703b0 msgr2=0x7fccb810cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:42.036 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:42.024+0000 7fccbec0c700 1 --2- 172.21.15.52:0/2090528890 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fccb80703b0 0x7fccb810cdd0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fccac00df90 tx=0x7fccac006ee0).stop 2024-02-19T22:08:42.036 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:42.024+0000 7fccbec0c700 1 -- 172.21.15.52:0/2090528890 shutdown_connections 2024-02-19T22:08:42.036 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:42.024+0000 7fccbec0c700 1 --2- 172.21.15.52:0/2090528890 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fccb80703b0 0x7fccb810cdd0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:42.036 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:42.024+0000 7fccbec0c700 1 -- 172.21.15.52:0/2090528890 >> 172.21.15.52:0/2090528890 conn(0x7fccb80f6530 msgr2=0x7fccb80f7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:42.036 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:42.024+0000 7fccbec0c700 1 -- 172.21.15.52:0/2090528890 shutdown_connections 2024-02-19T22:08:42.036 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:42.024+0000 7fccbec0c700 1 -- 172.21.15.52:0/2090528890 wait complete. 2024-02-19T22:08:42.341 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:42 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:08:42.024745+0000 mon.smithi052 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.52:0/2090528890' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-19T22:08:42.858 INFO:teuthology.orchestra.run.smithi052.stderr:mgr not available, waiting (1/15)... 2024-02-19T22:08:44.861 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi052 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphvym4zi3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp6ou7w4ul:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-02-19T22:08:45.488 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:45.486+0000 7fbc0f83e700 1 Processor -- start 2024-02-19T22:08:45.488 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:45.486+0000 7fbc0f83e700 1 -- start start 2024-02-19T22:08:45.490 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:45.486+0000 7fbc0f83e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc080703d0 0x7fbc08100eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:45.491 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:45.486+0000 7fbc0f83e700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc080fc8a0 con 0x7fbc080703d0 2024-02-19T22:08:45.491 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:45.487+0000 7fbc0d5da700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc080703d0 0x7fbc08100eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:45.491 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:45.487+0000 7fbc0d5da700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc080703d0 0x7fbc08100eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:38492/0 (socket says 172.21.15.52:38492) 2024-02-19T22:08:45.491 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:45.487+0000 7fbc0d5da700 1 -- 172.21.15.52:0/2761922663 learned_addr learned my addr 172.21.15.52:0/2761922663 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:08:45.491 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:45.487+0000 7fbc0d5da700 1 -- 172.21.15.52:0/2761922663 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc081013a0 con 0x7fbc080703d0 2024-02-19T22:08:45.491 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:45.487+0000 7fbc0d5da700 1 --2- 172.21.15.52:0/2761922663 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc080703d0 0x7fbc08100eb0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fbc0400b5f0 tx=0x7fbc04006ee0).ready entity=mon.0 client_cookie=136ed238539c036b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:45.491 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:45.487+0000 7fbbfbfff700 1 -- 172.21.15.52:0/2761922663 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc0400d600 con 0x7fbc080703d0 2024-02-19T22:08:45.491 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:45.487+0000 7fbbfbfff700 1 -- 172.21.15.52:0/2761922663 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fbc0400dbf0 con 0x7fbc080703d0 2024-02-19T22:08:45.491 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:45.487+0000 7fbbfbfff700 1 -- 172.21.15.52:0/2761922663 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc04011cb0 con 0x7fbc080703d0 2024-02-19T22:08:45.491 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:45.488+0000 7fbc0f83e700 1 -- 172.21.15.52:0/2761922663 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc080703d0 msgr2=0x7fbc08100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:45.491 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:45.488+0000 7fbc0f83e700 1 --2- 172.21.15.52:0/2761922663 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc080703d0 0x7fbc08100eb0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fbc0400b5f0 tx=0x7fbc04006ee0).stop 2024-02-19T22:08:45.491 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:45.488+0000 7fbc0f83e700 1 -- 172.21.15.52:0/2761922663 shutdown_connections 2024-02-19T22:08:45.491 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:45.488+0000 7fbc0f83e700 1 --2- 172.21.15.52:0/2761922663 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc080703d0 0x7fbc08100eb0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:45.492 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:45.488+0000 7fbc0f83e700 1 -- 172.21.15.52:0/2761922663 >> 172.21.15.52:0/2761922663 conn(0x7fbc080f6530 msgr2=0x7fbc080f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:45.492 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:45.488+0000 7fbc0f83e700 1 -- 172.21.15.52:0/2761922663 shutdown_connections 2024-02-19T22:08:45.492 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:45.488+0000 7fbc0f83e700 1 -- 172.21.15.52:0/2761922663 wait complete. 2024-02-19T22:08:45.492 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:45.489+0000 7fbc0f83e700 1 Processor -- start 2024-02-19T22:08:45.492 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:45.489+0000 7fbc0f83e700 1 -- start start 2024-02-19T22:08:45.492 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:45.489+0000 7fbc0f83e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc080703d0 0x7fbc0810ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:45.492 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:45.489+0000 7fbc0f83e700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0810d310 con 0x7fbc080703d0 2024-02-19T22:08:45.492 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:45.489+0000 7fbc0d5da700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc080703d0 0x7fbc0810ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:45.492 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:45.489+0000 7fbc0d5da700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc080703d0 0x7fbc0810ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:38494/0 (socket says 172.21.15.52:38494) 2024-02-19T22:08:45.492 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:45.489+0000 7fbc0d5da700 1 -- 172.21.15.52:0/2930546219 learned_addr learned my addr 172.21.15.52:0/2930546219 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:08:45.492 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:45.489+0000 7fbc0d5da700 1 -- 172.21.15.52:0/2930546219 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc0400b200 con 0x7fbc080703d0 2024-02-19T22:08:45.492 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:45.489+0000 7fbc0d5da700 1 --2- 172.21.15.52:0/2930546219 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc080703d0 0x7fbc0810ce20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fbc04006c70 tx=0x7fbc04006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:45.493 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:45.490+0000 7fbbf9ffb700 1 -- 172.21.15.52:0/2930546219 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc04011400 con 0x7fbc080703d0 2024-02-19T22:08:45.494 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:45.490+0000 7fbbf9ffb700 1 -- 172.21.15.52:0/2930546219 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fbc0401b8a0 con 0x7fbc080703d0 2024-02-19T22:08:45.494 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:45.490+0000 7fbbf9ffb700 1 -- 172.21.15.52:0/2930546219 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc0401a7c0 con 0x7fbc080703d0 2024-02-19T22:08:45.494 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:45.490+0000 7fbc0f83e700 1 -- 172.21.15.52:0/2930546219 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc0810d510 con 0x7fbc080703d0 2024-02-19T22:08:45.494 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:45.490+0000 7fbc0f83e700 1 -- 172.21.15.52:0/2930546219 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc0810e120 con 0x7fbc080703d0 2024-02-19T22:08:45.494 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:45.490+0000 7fbbf9ffb700 1 -- 172.21.15.52:0/2930546219 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fbc0400f070 con 0x7fbc080703d0 2024-02-19T22:08:45.494 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:45.490+0000 7fbbf9ffb700 1 -- 172.21.15.52:0/2930546219 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fbc04025b20 con 0x7fbc080703d0 2024-02-19T22:08:45.494 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:45.491+0000 7fbc0f83e700 1 -- 172.21.15.52:0/2930546219 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc08059100 con 0x7fbc080703d0 2024-02-19T22:08:45.494 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:45.493+0000 7fbbf9ffb700 1 -- 172.21.15.52:0/2930546219 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fbc04016070 con 0x7fbc080703d0 2024-02-19T22:08:45.565 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:45.564+0000 7fbc0f83e700 1 -- 172.21.15.52:0/2930546219 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fbc08102050 con 0x7fbc080703d0 2024-02-19T22:08:45.565 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:45.564+0000 7fbbf9ffb700 1 -- 172.21.15.52:0/2930546219 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7fbc04025d30 con 0x7fbc080703d0 2024-02-19T22:08:45.566 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:45.566 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: { 2024-02-19T22:08:45.570 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "fsid": "56739a54-cf73-11ee-95bb-87774f69a715", 2024-02-19T22:08:45.570 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "health": { 2024-02-19T22:08:45.570 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-19T22:08:45.570 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "checks": {}, 2024-02-19T22:08:45.570 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "mutes": [] 2024-02-19T22:08:45.570 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: }, 2024-02-19T22:08:45.570 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-19T22:08:45.570 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "quorum": [ 2024-02-19T22:08:45.570 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 0 2024-02-19T22:08:45.570 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: ], 2024-02-19T22:08:45.570 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-19T22:08:45.570 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "smithi052" 2024-02-19T22:08:45.571 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: ], 2024-02-19T22:08:45.571 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "quorum_age": 6, 2024-02-19T22:08:45.571 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "monmap": { 2024-02-19T22:08:45.571 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:08:45.571 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-19T22:08:45.571 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-19T22:08:45.571 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: }, 2024-02-19T22:08:45.571 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "osdmap": { 2024-02-19T22:08:45.571 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:08:45.571 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-19T22:08:45.571 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-19T22:08:45.571 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-19T22:08:45.571 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-19T22:08:45.571 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-19T22:08:45.571 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-19T22:08:45.571 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: }, 2024-02-19T22:08:45.572 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "pgmap": { 2024-02-19T22:08:45.572 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-19T22:08:45.572 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-19T22:08:45.572 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-19T22:08:45.572 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-19T22:08:45.572 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-19T22:08:45.572 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-19T22:08:45.572 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-19T22:08:45.575 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-19T22:08:45.575 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: }, 2024-02-19T22:08:45.575 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "fsmap": { 2024-02-19T22:08:45.575 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:08:45.575 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-19T22:08:45.575 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-19T22:08:45.575 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: }, 2024-02-19T22:08:45.575 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-19T22:08:45.575 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "available": false, 2024-02-19T22:08:45.576 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-19T22:08:45.576 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "modules": [ 2024-02-19T22:08:45.576 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "iostat", 2024-02-19T22:08:45.576 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "restful" 2024-02-19T22:08:45.576 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: ], 2024-02-19T22:08:45.576 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "services": {} 2024-02-19T22:08:45.576 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: }, 2024-02-19T22:08:45.576 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "servicemap": { 2024-02-19T22:08:45.576 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:08:45.576 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "modified": "2024-02-19T22:08:33.546528+0000", 2024-02-19T22:08:45.576 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "services": {} 2024-02-19T22:08:45.576 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: }, 2024-02-19T22:08:45.576 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-19T22:08:45.576 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: } 2024-02-19T22:08:45.576 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:45.565+0000 7fbc0f83e700 1 -- 172.21.15.52:0/2930546219 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc080703d0 msgr2=0x7fbc0810ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:45.577 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:45.565+0000 7fbc0f83e700 1 --2- 172.21.15.52:0/2930546219 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc080703d0 0x7fbc0810ce20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fbc04006c70 tx=0x7fbc04006ee0).stop 2024-02-19T22:08:45.577 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:45.565+0000 7fbc0f83e700 1 -- 172.21.15.52:0/2930546219 shutdown_connections 2024-02-19T22:08:45.577 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:45.565+0000 7fbc0f83e700 1 --2- 172.21.15.52:0/2930546219 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc080703d0 0x7fbc0810ce20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:45.577 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:45.565+0000 7fbc0f83e700 1 -- 172.21.15.52:0/2930546219 >> 172.21.15.52:0/2930546219 conn(0x7fbc080f6530 msgr2=0x7fbc080f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:45.577 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:45.565+0000 7fbc0f83e700 1 -- 172.21.15.52:0/2930546219 shutdown_connections 2024-02-19T22:08:45.577 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:45.565+0000 7fbc0f83e700 1 -- 172.21.15.52:0/2930546219 wait complete. 2024-02-19T22:08:45.945 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:45 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:08:45.565712+0000 mon.smithi052 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.52:0/2930546219' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-19T22:08:46.365 INFO:teuthology.orchestra.run.smithi052.stderr:mgr not available, waiting (2/15)... 2024-02-19T22:08:46.842 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:08:46.485720+0000 mon.smithi052 (mon.0) 9 : cluster [INF] Activating manager daemon smithi052.ghqrtk 2024-02-19T22:08:46.842 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:08:46.488103+0000 mon.smithi052 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi052.ghqrtk(active, starting, since 0.0024766s) 2024-02-19T22:08:46.842 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:08:46.488957+0000 mon.smithi052 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.52:0/222523688' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T22:08:46.842 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:08:46.489176+0000 mon.smithi052 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.52:0/222523688' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T22:08:46.842 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:08:46.489330+0000 mon.smithi052 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.52:0/222523688' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T22:08:46.842 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:08:46.489450+0000 mon.smithi052 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.52:0/222523688' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix": "mon metadata", "id": "smithi052"}]: dispatch 2024-02-19T22:08:46.842 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:08:46.489975+0000 mon.smithi052 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.52:0/222523688' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix": "mgr metadata", "who": "smithi052.ghqrtk", "id": "smithi052.ghqrtk"}]: dispatch 2024-02-19T22:08:46.842 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:08:46.503621+0000 mon.smithi052 (mon.0) 16 : cluster [INF] Manager daemon smithi052.ghqrtk is now available 2024-02-19T22:08:46.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:08:46.510849+0000 mon.smithi052 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.52:0/222523688' entity='mgr.smithi052.ghqrtk' 2024-02-19T22:08:46.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:08:46.515925+0000 mon.smithi052 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.52:0/222523688' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi052.ghqrtk/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:08:46.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:08:46.517019+0000 mon.smithi052 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.52:0/222523688' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi052.ghqrtk/trash_purge_schedule"}]: dispatch 2024-02-19T22:08:46.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:08:46.521864+0000 mon.smithi052 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.52:0/222523688' entity='mgr.smithi052.ghqrtk' 2024-02-19T22:08:46.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:08:46.523649+0000 mon.smithi052 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.52:0/222523688' entity='mgr.smithi052.ghqrtk' 2024-02-19T22:08:48.367 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi052 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphvym4zi3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp6ou7w4ul:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-02-19T22:08:48.841 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:48 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:08:47.491550+0000 mon.smithi052 (mon.0) 22 : cluster [DBG] mgrmap e3: smithi052.ghqrtk(active, since 1.00592s) 2024-02-19T22:08:48.955 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:48.953+0000 7f442bb51700 1 Processor -- start 2024-02-19T22:08:48.956 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:48.954+0000 7f442bb51700 1 -- start start 2024-02-19T22:08:48.956 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:48.954+0000 7f442bb51700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44240703f0 0x7f44240fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:48.956 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:48.954+0000 7f442bb51700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44240fef80 con 0x7f44240703f0 2024-02-19T22:08:48.956 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:48.954+0000 7f44298ed700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44240703f0 0x7f44240fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:48.956 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:48.954+0000 7f44298ed700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44240703f0 0x7f44240fea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:38538/0 (socket says 172.21.15.52:38538) 2024-02-19T22:08:48.956 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:48.954+0000 7f44298ed700 1 -- 172.21.15.52:0/2673120775 learned_addr learned my addr 172.21.15.52:0/2673120775 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:08:48.956 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:48.955+0000 7f44298ed700 1 -- 172.21.15.52:0/2673120775 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44240ff0c0 con 0x7f44240703f0 2024-02-19T22:08:48.959 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:48.955+0000 7f44298ed700 1 --2- 172.21.15.52:0/2673120775 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44240703f0 0x7f44240fea90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f442000b5f0 tx=0x7f4420006ee0).ready entity=mon.0 client_cookie=eaa41c194dd4ad0b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:48.959 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:48.955+0000 7f44288eb700 1 -- 172.21.15.52:0/2673120775 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f442000d600 con 0x7f44240703f0 2024-02-19T22:08:48.959 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:48.955+0000 7f44288eb700 1 -- 172.21.15.52:0/2673120775 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f442000dbf0 con 0x7f44240703f0 2024-02-19T22:08:48.959 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:48.955+0000 7f44288eb700 1 -- 172.21.15.52:0/2673120775 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4420011ce0 con 0x7f44240703f0 2024-02-19T22:08:48.959 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:48.956+0000 7f442bb51700 1 -- 172.21.15.52:0/2673120775 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44240703f0 msgr2=0x7f44240fea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:48.959 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:48.956+0000 7f442bb51700 1 --2- 172.21.15.52:0/2673120775 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44240703f0 0x7f44240fea90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f442000b5f0 tx=0x7f4420006ee0).stop 2024-02-19T22:08:48.960 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:48.956+0000 7f442bb51700 1 -- 172.21.15.52:0/2673120775 shutdown_connections 2024-02-19T22:08:48.960 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:48.956+0000 7f442bb51700 1 --2- 172.21.15.52:0/2673120775 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44240703f0 0x7f44240fea90 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:48.960 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:48.956+0000 7f442bb51700 1 -- 172.21.15.52:0/2673120775 >> 172.21.15.52:0/2673120775 conn(0x7f44240f6530 msgr2=0x7f44240f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:48.960 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:48.956+0000 7f442bb51700 1 -- 172.21.15.52:0/2673120775 shutdown_connections 2024-02-19T22:08:48.960 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:48.956+0000 7f442bb51700 1 -- 172.21.15.52:0/2673120775 wait complete. 2024-02-19T22:08:48.960 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:48.956+0000 7f442bb51700 1 Processor -- start 2024-02-19T22:08:48.960 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:48.956+0000 7f442bb51700 1 -- start start 2024-02-19T22:08:48.960 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:48.957+0000 7f442bb51700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44240703f0 0x7f44240fc510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:48.960 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:48.957+0000 7f442bb51700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44240fa990 con 0x7f44240703f0 2024-02-19T22:08:48.960 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:48.957+0000 7f44298ed700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44240703f0 0x7f44240fc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:48.960 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:48.957+0000 7f44298ed700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44240703f0 0x7f44240fc510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:38552/0 (socket says 172.21.15.52:38552) 2024-02-19T22:08:48.960 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:48.957+0000 7f44298ed700 1 -- 172.21.15.52:0/3221853197 learned_addr learned my addr 172.21.15.52:0/3221853197 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:08:48.960 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:48.957+0000 7f44298ed700 1 -- 172.21.15.52:0/3221853197 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f442000b200 con 0x7f44240703f0 2024-02-19T22:08:48.960 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:48.957+0000 7f44298ed700 1 --2- 172.21.15.52:0/3221853197 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44240703f0 0x7f44240fc510 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f44200115c0 tx=0x7f4420006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:48.960 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:48.957+0000 7f44167fc700 1 -- 172.21.15.52:0/3221853197 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f442001b410 con 0x7f44240703f0 2024-02-19T22:08:48.961 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:48.957+0000 7f442bb51700 1 -- 172.21.15.52:0/3221853197 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44240fab30 con 0x7f44240703f0 2024-02-19T22:08:48.961 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:48.957+0000 7f44167fc700 1 -- 172.21.15.52:0/3221853197 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f442001ba00 con 0x7f44240703f0 2024-02-19T22:08:48.961 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:48.957+0000 7f44167fc700 1 -- 172.21.15.52:0/3221853197 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f442001aaa0 con 0x7f44240703f0 2024-02-19T22:08:48.961 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:48.957+0000 7f442bb51700 1 -- 172.21.15.52:0/3221853197 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44240fafc0 con 0x7f44240703f0 2024-02-19T22:08:48.961 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:48.958+0000 7f44167fc700 1 -- 172.21.15.52:0/3221853197 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 3) v1 ==== 36029+0+0 (secure 0 0 0) 0x7f442000f070 con 0x7f44240703f0 2024-02-19T22:08:48.961 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:48.958+0000 7f44167fc700 1 --2- 172.21.15.52:0/3221853197 >> [v2:172.21.15.52:6800/384714397,v1:172.21.15.52:6801/384714397] conn(0x7f440c02fd10 0x7f440c0321d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:48.961 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:48.958+0000 7f442bb51700 1 -- 172.21.15.52:0/3221853197 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4424045ad0 con 0x7f44240703f0 2024-02-19T22:08:48.962 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:48.958+0000 7f44167fc700 1 -- 172.21.15.52:0/3221853197 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f4420040ee0 con 0x7f44240703f0 2024-02-19T22:08:48.962 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:48.958+0000 7f44290ec700 1 --2- 172.21.15.52:0/3221853197 >> [v2:172.21.15.52:6800/384714397,v1:172.21.15.52:6801/384714397] conn(0x7f440c02fd10 0x7f440c0321d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:48.962 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:48.959+0000 7f44290ec700 1 --2- 172.21.15.52:0/3221853197 >> [v2:172.21.15.52:6800/384714397,v1:172.21.15.52:6801/384714397] conn(0x7f440c02fd10 0x7f440c0321d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f44180096a0 tx=0x7f4418006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:48.964 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:48.963+0000 7f44167fc700 1 -- 172.21.15.52:0/3221853197 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4420016070 con 0x7f44240703f0 2024-02-19T22:08:49.186 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:49.179+0000 7f442bb51700 1 -- 172.21.15.52:0/3221853197 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f4424027a00 con 0x7f44240703f0 2024-02-19T22:08:49.186 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:49.179+0000 7f44167fc700 1 -- 172.21.15.52:0/3221853197 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f4420010360 con 0x7f44240703f0 2024-02-19T22:08:49.186 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:49.187 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: { 2024-02-19T22:08:49.187 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "fsid": "56739a54-cf73-11ee-95bb-87774f69a715", 2024-02-19T22:08:49.187 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "health": { 2024-02-19T22:08:49.187 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-19T22:08:49.187 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "checks": {}, 2024-02-19T22:08:49.187 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "mutes": [] 2024-02-19T22:08:49.187 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: }, 2024-02-19T22:08:49.187 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-19T22:08:49.187 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "quorum": [ 2024-02-19T22:08:49.187 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 0 2024-02-19T22:08:49.187 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: ], 2024-02-19T22:08:49.187 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-19T22:08:49.187 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "smithi052" 2024-02-19T22:08:49.187 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: ], 2024-02-19T22:08:49.187 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "quorum_age": 10, 2024-02-19T22:08:49.188 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "monmap": { 2024-02-19T22:08:49.188 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:08:49.188 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-19T22:08:49.188 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-19T22:08:49.188 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: }, 2024-02-19T22:08:49.188 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "osdmap": { 2024-02-19T22:08:49.188 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:08:49.188 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-19T22:08:49.188 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-19T22:08:49.188 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-19T22:08:49.188 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-19T22:08:49.188 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-19T22:08:49.188 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-19T22:08:49.188 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: }, 2024-02-19T22:08:49.188 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "pgmap": { 2024-02-19T22:08:49.189 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-19T22:08:49.189 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-19T22:08:49.189 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-19T22:08:49.189 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-19T22:08:49.189 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-19T22:08:49.189 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-19T22:08:49.189 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-19T22:08:49.189 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-19T22:08:49.189 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: }, 2024-02-19T22:08:49.189 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "fsmap": { 2024-02-19T22:08:49.189 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:08:49.189 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-19T22:08:49.189 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-19T22:08:49.189 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: }, 2024-02-19T22:08:49.189 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-19T22:08:49.190 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "available": true, 2024-02-19T22:08:49.190 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-19T22:08:49.190 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "modules": [ 2024-02-19T22:08:49.190 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "iostat", 2024-02-19T22:08:49.190 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "restful" 2024-02-19T22:08:49.190 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: ], 2024-02-19T22:08:49.190 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "services": {} 2024-02-19T22:08:49.190 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: }, 2024-02-19T22:08:49.190 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "servicemap": { 2024-02-19T22:08:49.190 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:08:49.190 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "modified": "2024-02-19T22:08:33.546528+0000", 2024-02-19T22:08:49.190 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "services": {} 2024-02-19T22:08:49.190 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: }, 2024-02-19T22:08:49.190 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-19T22:08:49.190 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: } 2024-02-19T22:08:49.191 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:49.181+0000 7f442bb51700 1 -- 172.21.15.52:0/3221853197 >> [v2:172.21.15.52:6800/384714397,v1:172.21.15.52:6801/384714397] conn(0x7f440c02fd10 msgr2=0x7f440c0321d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:49.191 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:49.181+0000 7f442bb51700 1 --2- 172.21.15.52:0/3221853197 >> [v2:172.21.15.52:6800/384714397,v1:172.21.15.52:6801/384714397] conn(0x7f440c02fd10 0x7f440c0321d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f44180096a0 tx=0x7f4418006b40).stop 2024-02-19T22:08:49.191 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:49.181+0000 7f442bb51700 1 -- 172.21.15.52:0/3221853197 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44240703f0 msgr2=0x7f44240fc510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:49.191 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:49.181+0000 7f442bb51700 1 --2- 172.21.15.52:0/3221853197 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44240703f0 0x7f44240fc510 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f44200115c0 tx=0x7f4420006ee0).stop 2024-02-19T22:08:49.191 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:49.181+0000 7f442bb51700 1 -- 172.21.15.52:0/3221853197 shutdown_connections 2024-02-19T22:08:49.191 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:49.181+0000 7f442bb51700 1 --2- 172.21.15.52:0/3221853197 >> [v2:172.21.15.52:6800/384714397,v1:172.21.15.52:6801/384714397] conn(0x7f440c02fd10 0x7f440c0321d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:49.191 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:49.181+0000 7f442bb51700 1 --2- 172.21.15.52:0/3221853197 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44240703f0 0x7f44240fc510 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:49.191 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:49.181+0000 7f442bb51700 1 -- 172.21.15.52:0/3221853197 >> 172.21.15.52:0/3221853197 conn(0x7f44240f6530 msgr2=0x7f44240f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:49.191 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:49.181+0000 7f442bb51700 1 -- 172.21.15.52:0/3221853197 shutdown_connections 2024-02-19T22:08:49.191 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:49.181+0000 7f442bb51700 1 -- 172.21.15.52:0/3221853197 wait complete. 2024-02-19T22:08:50.342 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:50 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:08:49.180681+0000 mon.smithi052 (mon.0) 23 : audit [DBG] from='client.? 172.21.15.52:0/3221853197' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-19T22:08:50.342 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:50 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:08:49.491097+0000 mon.smithi052 (mon.0) 24 : cluster [DBG] mgrmap e4: smithi052.ghqrtk(active, since 3s) 2024-02-19T22:08:53.891 INFO:teuthology.orchestra.run.smithi052.stderr:mgr is available 2024-02-19T22:08:53.891 INFO:teuthology.orchestra.run.smithi052.stderr:Enabling cephadm module... 2024-02-19T22:08:53.891 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi052 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphvym4zi3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp6ou7w4ul:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2024-02-19T22:08:55.585 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.581+0000 7fcc56ee3700 1 Processor -- start 2024-02-19T22:08:55.585 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.582+0000 7fcc56ee3700 1 -- start start 2024-02-19T22:08:55.585 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.582+0000 7fcc56ee3700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcc500703f0 0x7fcc500fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:55.585 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.582+0000 7fcc56ee3700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc500fef80 con 0x7fcc500703f0 2024-02-19T22:08:55.585 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.582+0000 7fcc54c7f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcc500703f0 0x7fcc500fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:55.585 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.582+0000 7fcc54c7f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcc500703f0 0x7fcc500fea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51380/0 (socket says 172.21.15.52:51380) 2024-02-19T22:08:55.585 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.582+0000 7fcc54c7f700 1 -- 172.21.15.52:0/3333121301 learned_addr learned my addr 172.21.15.52:0/3333121301 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:08:55.585 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.582+0000 7fcc54c7f700 1 -- 172.21.15.52:0/3333121301 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc500ff0c0 con 0x7fcc500703f0 2024-02-19T22:08:55.585 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.582+0000 7fcc54c7f700 1 --2- 172.21.15.52:0/3333121301 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcc500703f0 0x7fcc500fea90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fcc4400b5f0 tx=0x7fcc44006ee0).ready entity=mon.0 client_cookie=3f7ab1e3227f9aad server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:55.585 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.582+0000 7fcc4f7fe700 1 -- 172.21.15.52:0/3333121301 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc4400d600 con 0x7fcc500703f0 2024-02-19T22:08:55.585 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.583+0000 7fcc4f7fe700 1 -- 172.21.15.52:0/3333121301 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fcc4400dbf0 con 0x7fcc500703f0 2024-02-19T22:08:55.585 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.583+0000 7fcc4f7fe700 1 -- 172.21.15.52:0/3333121301 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc44011ce0 con 0x7fcc500703f0 2024-02-19T22:08:55.585 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.583+0000 7fcc56ee3700 1 -- 172.21.15.52:0/3333121301 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcc500703f0 msgr2=0x7fcc500fea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:55.586 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.583+0000 7fcc56ee3700 1 --2- 172.21.15.52:0/3333121301 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcc500703f0 0x7fcc500fea90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fcc4400b5f0 tx=0x7fcc44006ee0).stop 2024-02-19T22:08:55.586 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.583+0000 7fcc56ee3700 1 -- 172.21.15.52:0/3333121301 shutdown_connections 2024-02-19T22:08:55.586 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.583+0000 7fcc56ee3700 1 --2- 172.21.15.52:0/3333121301 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcc500703f0 0x7fcc500fea90 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:55.586 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.583+0000 7fcc56ee3700 1 -- 172.21.15.52:0/3333121301 >> 172.21.15.52:0/3333121301 conn(0x7fcc500f6530 msgr2=0x7fcc500f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:55.586 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.584+0000 7fcc56ee3700 1 -- 172.21.15.52:0/3333121301 shutdown_connections 2024-02-19T22:08:55.586 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.584+0000 7fcc56ee3700 1 -- 172.21.15.52:0/3333121301 wait complete. 2024-02-19T22:08:55.588 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.584+0000 7fcc56ee3700 1 Processor -- start 2024-02-19T22:08:55.588 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.584+0000 7fcc56ee3700 1 -- start start 2024-02-19T22:08:55.588 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.585+0000 7fcc56ee3700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcc500703f0 0x7fcc5010ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:55.589 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.585+0000 7fcc56ee3700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5010b010 con 0x7fcc500703f0 2024-02-19T22:08:55.589 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.585+0000 7fcc54c7f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcc500703f0 0x7fcc5010ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:55.589 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.585+0000 7fcc54c7f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcc500703f0 0x7fcc5010ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51388/0 (socket says 172.21.15.52:51388) 2024-02-19T22:08:55.589 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.585+0000 7fcc54c7f700 1 -- 172.21.15.52:0/637420232 learned_addr learned my addr 172.21.15.52:0/637420232 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:08:55.589 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.585+0000 7fcc54c7f700 1 -- 172.21.15.52:0/637420232 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc4400b200 con 0x7fcc500703f0 2024-02-19T22:08:55.589 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.585+0000 7fcc54c7f700 1 --2- 172.21.15.52:0/637420232 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcc500703f0 0x7fcc5010ab20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fcc44011ed0 tx=0x7fcc44006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:55.589 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.585+0000 7fcc4d7fa700 1 -- 172.21.15.52:0/637420232 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc4401b570 con 0x7fcc500703f0 2024-02-19T22:08:55.589 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.585+0000 7fcc4d7fa700 1 -- 172.21.15.52:0/637420232 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fcc4401bb60 con 0x7fcc500703f0 2024-02-19T22:08:55.589 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.585+0000 7fcc56ee3700 1 -- 172.21.15.52:0/637420232 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc5010b210 con 0x7fcc500703f0 2024-02-19T22:08:55.589 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.585+0000 7fcc4d7fa700 1 -- 172.21.15.52:0/637420232 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc4401aad0 con 0x7fcc500703f0 2024-02-19T22:08:55.589 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.585+0000 7fcc56ee3700 1 -- 172.21.15.52:0/637420232 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc5010b6a0 con 0x7fcc500703f0 2024-02-19T22:08:55.589 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.586+0000 7fcc4d7fa700 1 -- 172.21.15.52:0/637420232 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7fcc4400f070 con 0x7fcc500703f0 2024-02-19T22:08:55.589 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.586+0000 7fcc4d7fa700 1 --2- 172.21.15.52:0/637420232 >> [v2:172.21.15.52:6800/384714397,v1:172.21.15.52:6801/384714397] conn(0x7fcc3402fe10 0x7fcc340322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:55.589 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.586+0000 7fcc4d7fa700 1 -- 172.21.15.52:0/637420232 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fcc44042130 con 0x7fcc500703f0 2024-02-19T22:08:55.589 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.586+0000 7fcc56ee3700 1 -- 172.21.15.52:0/637420232 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc50045ad0 con 0x7fcc500703f0 2024-02-19T22:08:55.590 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.586+0000 7fcc4ffff700 1 --2- 172.21.15.52:0/637420232 >> [v2:172.21.15.52:6800/384714397,v1:172.21.15.52:6801/384714397] conn(0x7fcc3402fe10 0x7fcc340322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:55.590 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.587+0000 7fcc4ffff700 1 --2- 172.21.15.52:0/637420232 >> [v2:172.21.15.52:6800/384714397,v1:172.21.15.52:6801/384714397] conn(0x7fcc3402fe10 0x7fcc340322d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fcc3c0096a0 tx=0x7fcc3c006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:55.592 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.591+0000 7fcc4d7fa700 1 -- 172.21.15.52:0/637420232 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcc44016070 con 0x7fcc500703f0 2024-02-19T22:08:55.788 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.787+0000 7fcc56ee3700 1 -- 172.21.15.52:0/637420232 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fcc500ffe00 con 0x7fcc500703f0 2024-02-19T22:08:55.841 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.840+0000 7fcc4d7fa700 1 -- 172.21.15.52:0/637420232 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fcc4401bcd0 con 0x7fcc500703f0 2024-02-19T22:08:55.842 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.841+0000 7fcc4d7fa700 1 -- 172.21.15.52:0/637420232 <== mon.0 v2:172.21.15.52:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7fcc44010be0 con 0x7fcc500703f0 2024-02-19T22:08:55.845 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.843+0000 7fcc56ee3700 1 -- 172.21.15.52:0/637420232 >> [v2:172.21.15.52:6800/384714397,v1:172.21.15.52:6801/384714397] conn(0x7fcc3402fe10 msgr2=0x7fcc340322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:55.845 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.843+0000 7fcc56ee3700 1 --2- 172.21.15.52:0/637420232 >> [v2:172.21.15.52:6800/384714397,v1:172.21.15.52:6801/384714397] conn(0x7fcc3402fe10 0x7fcc340322d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fcc3c0096a0 tx=0x7fcc3c006b40).stop 2024-02-19T22:08:55.845 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.843+0000 7fcc56ee3700 1 -- 172.21.15.52:0/637420232 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcc500703f0 msgr2=0x7fcc5010ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:55.845 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.843+0000 7fcc56ee3700 1 --2- 172.21.15.52:0/637420232 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcc500703f0 0x7fcc5010ab20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fcc44011ed0 tx=0x7fcc44006ee0).stop 2024-02-19T22:08:55.845 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.843+0000 7fcc56ee3700 1 -- 172.21.15.52:0/637420232 shutdown_connections 2024-02-19T22:08:55.845 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.843+0000 7fcc56ee3700 1 --2- 172.21.15.52:0/637420232 >> [v2:172.21.15.52:6800/384714397,v1:172.21.15.52:6801/384714397] conn(0x7fcc3402fe10 0x7fcc340322d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:55.845 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.843+0000 7fcc56ee3700 1 --2- 172.21.15.52:0/637420232 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcc500703f0 0x7fcc5010ab20 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:55.845 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.843+0000 7fcc56ee3700 1 -- 172.21.15.52:0/637420232 >> 172.21.15.52:0/637420232 conn(0x7fcc500f6530 msgr2=0x7fcc500f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:55.845 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.843+0000 7fcc56ee3700 1 -- 172.21.15.52:0/637420232 shutdown_connections 2024-02-19T22:08:55.845 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:55.843+0000 7fcc56ee3700 1 -- 172.21.15.52:0/637420232 wait complete. 2024-02-19T22:08:56.203 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:55 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:08:55.789190+0000 mon.smithi052 (mon.0) 25 : audit [INF] from='client.? 172.21.15.52:0/637420232' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-19T22:08:56.472 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi052 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphvym4zi3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp6ou7w4ul:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-02-19T22:08:57.091 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:08:55.842381+0000 mon.smithi052 (mon.0) 26 : audit [INF] from='client.? 172.21.15.52:0/637420232' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-19T22:08:57.091 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:08:55.842433+0000 mon.smithi052 (mon.0) 27 : cluster [DBG] mgrmap e5: smithi052.ghqrtk(active, since 9s) 2024-02-19T22:08:57.184 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.181+0000 7f3d9526c700 1 Processor -- start 2024-02-19T22:08:57.184 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.181+0000 7f3d9526c700 1 -- start start 2024-02-19T22:08:57.184 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.182+0000 7f3d9526c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3d900703f0 0x7f3d900fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:57.184 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.182+0000 7f3d9526c700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d900fef80 con 0x7f3d900703f0 2024-02-19T22:08:57.184 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.182+0000 7f3d8effd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3d900703f0 0x7f3d900fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:57.184 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.182+0000 7f3d8effd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3d900703f0 0x7f3d900fea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51418/0 (socket says 172.21.15.52:51418) 2024-02-19T22:08:57.184 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.182+0000 7f3d8effd700 1 -- 172.21.15.52:0/2527292879 learned_addr learned my addr 172.21.15.52:0/2527292879 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:08:57.184 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.182+0000 7f3d8effd700 1 -- 172.21.15.52:0/2527292879 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d900ff0c0 con 0x7f3d900703f0 2024-02-19T22:08:57.184 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.182+0000 7f3d8effd700 1 --2- 172.21.15.52:0/2527292879 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3d900703f0 0x7f3d900fea90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f3d8000b5f0 tx=0x7f3d80006ee0).ready entity=mon.0 client_cookie=6bcc0bca9305beda server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:57.184 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.183+0000 7f3d8dffb700 1 -- 172.21.15.52:0/2527292879 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d8000d600 con 0x7f3d900703f0 2024-02-19T22:08:57.184 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.183+0000 7f3d8dffb700 1 -- 172.21.15.52:0/2527292879 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f3d8000dbf0 con 0x7f3d900703f0 2024-02-19T22:08:57.186 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.183+0000 7f3d8dffb700 1 -- 172.21.15.52:0/2527292879 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d80011ce0 con 0x7f3d900703f0 2024-02-19T22:08:57.187 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.183+0000 7f3d9526c700 1 -- 172.21.15.52:0/2527292879 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3d900703f0 msgr2=0x7f3d900fea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:57.187 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.183+0000 7f3d9526c700 1 --2- 172.21.15.52:0/2527292879 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3d900703f0 0x7f3d900fea90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f3d8000b5f0 tx=0x7f3d80006ee0).stop 2024-02-19T22:08:57.187 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.183+0000 7f3d9526c700 1 -- 172.21.15.52:0/2527292879 shutdown_connections 2024-02-19T22:08:57.187 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.183+0000 7f3d9526c700 1 --2- 172.21.15.52:0/2527292879 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3d900703f0 0x7f3d900fea90 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:57.187 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.183+0000 7f3d9526c700 1 -- 172.21.15.52:0/2527292879 >> 172.21.15.52:0/2527292879 conn(0x7f3d900f6530 msgr2=0x7f3d900f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:57.187 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.183+0000 7f3d9526c700 1 -- 172.21.15.52:0/2527292879 shutdown_connections 2024-02-19T22:08:57.187 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.183+0000 7f3d9526c700 1 -- 172.21.15.52:0/2527292879 wait complete. 2024-02-19T22:08:57.187 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.184+0000 7f3d9526c700 1 Processor -- start 2024-02-19T22:08:57.187 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.184+0000 7f3d9526c700 1 -- start start 2024-02-19T22:08:57.187 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.184+0000 7f3d9526c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3d900703f0 0x7f3d90100660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:57.187 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.184+0000 7f3d9526c700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d90100b50 con 0x7f3d900703f0 2024-02-19T22:08:57.187 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.184+0000 7f3d8effd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3d900703f0 0x7f3d90100660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:57.187 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.184+0000 7f3d8effd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3d900703f0 0x7f3d90100660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51420/0 (socket says 172.21.15.52:51420) 2024-02-19T22:08:57.187 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.184+0000 7f3d8effd700 1 -- 172.21.15.52:0/1827921250 learned_addr learned my addr 172.21.15.52:0/1827921250 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:08:57.188 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.185+0000 7f3d8effd700 1 -- 172.21.15.52:0/1827921250 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d8000b200 con 0x7f3d900703f0 2024-02-19T22:08:57.188 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.185+0000 7f3d8effd700 1 --2- 172.21.15.52:0/1827921250 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3d900703f0 0x7f3d90100660 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f3d80011ed0 tx=0x7f3d80006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:57.188 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.185+0000 7f3d777fe700 1 -- 172.21.15.52:0/1827921250 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d8001b570 con 0x7f3d900703f0 2024-02-19T22:08:57.188 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.185+0000 7f3d9526c700 1 -- 172.21.15.52:0/1827921250 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d90100d50 con 0x7f3d900703f0 2024-02-19T22:08:57.188 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.185+0000 7f3d9526c700 1 -- 172.21.15.52:0/1827921250 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d901011e0 con 0x7f3d900703f0 2024-02-19T22:08:57.188 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.185+0000 7f3d777fe700 1 -- 172.21.15.52:0/1827921250 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f3d8001bb60 con 0x7f3d900703f0 2024-02-19T22:08:57.188 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.185+0000 7f3d777fe700 1 -- 172.21.15.52:0/1827921250 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d8001aad0 con 0x7f3d900703f0 2024-02-19T22:08:57.189 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.186+0000 7f3d777fe700 1 -- 172.21.15.52:0/1827921250 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f3d8000f070 con 0x7f3d900703f0 2024-02-19T22:08:57.189 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.186+0000 7f3d777fe700 1 --2- 172.21.15.52:0/1827921250 >> [v2:172.21.15.52:6800/384714397,v1:172.21.15.52:6801/384714397] conn(0x7f3d7c02fe70 0x7f3d7c032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:57.189 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.186+0000 7f3d777fe700 1 -- 172.21.15.52:0/1827921250 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f3d80042230 con 0x7f3d900703f0 2024-02-19T22:08:57.189 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.186+0000 7f3d8e7fc700 1 -- 172.21.15.52:0/1827921250 >> [v2:172.21.15.52:6800/384714397,v1:172.21.15.52:6801/384714397] conn(0x7f3d7c02fe70 msgr2=0x7f3d7c032330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/384714397 2024-02-19T22:08:57.189 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.186+0000 7f3d8e7fc700 1 --2- 172.21.15.52:0/1827921250 >> [v2:172.21.15.52:6800/384714397,v1:172.21.15.52:6801/384714397] conn(0x7f3d7c02fe70 0x7f3d7c032330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-19T22:08:57.189 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.187+0000 7f3d9526c700 1 -- 172.21.15.52:0/1827921250 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d90045ad0 con 0x7f3d900703f0 2024-02-19T22:08:57.192 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.191+0000 7f3d777fe700 1 -- 172.21.15.52:0/1827921250 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3d80016070 con 0x7f3d900703f0 2024-02-19T22:08:57.388 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.387+0000 7f3d8e7fc700 1 -- 172.21.15.52:0/1827921250 >> [v2:172.21.15.52:6800/384714397,v1:172.21.15.52:6801/384714397] conn(0x7f3d7c02fe70 msgr2=0x7f3d7c032330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/384714397 2024-02-19T22:08:57.388 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.387+0000 7f3d8e7fc700 1 --2- 172.21.15.52:0/1827921250 >> [v2:172.21.15.52:6800/384714397,v1:172.21.15.52:6801/384714397] conn(0x7f3d7c02fe70 0x7f3d7c032330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-19T22:08:57.395 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.393+0000 7f3d9526c700 1 -- 172.21.15.52:0/1827921250 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f3d90063aa0 con 0x7f3d900703f0 2024-02-19T22:08:57.395 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.393+0000 7f3d777fe700 1 -- 172.21.15.52:0/1827921250 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f3d80010360 con 0x7f3d900703f0 2024-02-19T22:08:57.395 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: { 2024-02-19T22:08:57.395 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "epoch": 5, 2024-02-19T22:08:57.395 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "available": true, 2024-02-19T22:08:57.395 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "active_name": "smithi052.ghqrtk", 2024-02-19T22:08:57.395 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-19T22:08:57.395 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: } 2024-02-19T22:08:57.397 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.395+0000 7f3d9526c700 1 -- 172.21.15.52:0/1827921250 >> [v2:172.21.15.52:6800/384714397,v1:172.21.15.52:6801/384714397] conn(0x7f3d7c02fe70 msgr2=0x7f3d7c032330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T22:08:57.397 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.395+0000 7f3d9526c700 1 --2- 172.21.15.52:0/1827921250 >> [v2:172.21.15.52:6800/384714397,v1:172.21.15.52:6801/384714397] conn(0x7f3d7c02fe70 0x7f3d7c032330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T22:08:57.397 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.395+0000 7f3d9526c700 1 -- 172.21.15.52:0/1827921250 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3d900703f0 msgr2=0x7f3d90100660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:57.397 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.395+0000 7f3d9526c700 1 --2- 172.21.15.52:0/1827921250 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3d900703f0 0x7f3d90100660 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f3d80011ed0 tx=0x7f3d80006ee0).stop 2024-02-19T22:08:57.398 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.395+0000 7f3d9526c700 1 -- 172.21.15.52:0/1827921250 shutdown_connections 2024-02-19T22:08:57.398 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.395+0000 7f3d9526c700 1 --2- 172.21.15.52:0/1827921250 >> [v2:172.21.15.52:6800/384714397,v1:172.21.15.52:6801/384714397] conn(0x7f3d7c02fe70 0x7f3d7c032330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T22:08:57.398 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.395+0000 7f3d9526c700 1 --2- 172.21.15.52:0/1827921250 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3d900703f0 0x7f3d90100660 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:57.398 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.395+0000 7f3d9526c700 1 -- 172.21.15.52:0/1827921250 >> 172.21.15.52:0/1827921250 conn(0x7f3d900f6530 msgr2=0x7f3d900fbc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:57.398 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.395+0000 7f3d9526c700 1 -- 172.21.15.52:0/1827921250 shutdown_connections 2024-02-19T22:08:57.398 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:57.395+0000 7f3d9526c700 1 -- 172.21.15.52:0/1827921250 wait complete. 2024-02-19T22:08:58.092 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:08:57 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:08:57.394875+0000 mon.smithi052 (mon.0) 28 : audit [DBG] from='client.? 172.21.15.52:0/1827921250' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-19T22:08:58.138 INFO:teuthology.orchestra.run.smithi052.stderr:Waiting for the mgr to restart... 2024-02-19T22:08:58.139 INFO:teuthology.orchestra.run.smithi052.stderr:Waiting for mgr epoch 5... 2024-02-19T22:08:58.139 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi052 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphvym4zi3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp6ou7w4ul:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-02-19T22:08:58.778 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:58.775+0000 7ff1cfbd4700 1 Processor -- start 2024-02-19T22:08:58.779 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:58.775+0000 7ff1cfbd4700 1 -- start start 2024-02-19T22:08:58.779 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:58.775+0000 7ff1cfbd4700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff1c80fad90 0x7ff1c80fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:58.779 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:58.775+0000 7ff1cfbd4700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c80fb6a0 con 0x7ff1c80fad90 2024-02-19T22:08:58.779 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:58.775+0000 7ff1cd970700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff1c80fad90 0x7ff1c80fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:58.779 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:58.776+0000 7ff1cd970700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff1c80fad90 0x7ff1c80fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51430/0 (socket says 172.21.15.52:51430) 2024-02-19T22:08:58.779 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:58.776+0000 7ff1cd970700 1 -- 172.21.15.52:0/971498184 learned_addr learned my addr 172.21.15.52:0/971498184 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:08:58.779 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:58.776+0000 7ff1cd970700 1 -- 172.21.15.52:0/971498184 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c80fb7e0 con 0x7ff1c80fad90 2024-02-19T22:08:58.779 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:58.776+0000 7ff1cd970700 1 --2- 172.21.15.52:0/971498184 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff1c80fad90 0x7ff1c80fb1b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7ff1c400b870 tx=0x7ff1c4006f70).ready entity=mon.0 client_cookie=4858d0255c306a80 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:58.779 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:58.776+0000 7ff1cc96e700 1 -- 172.21.15.52:0/971498184 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1c400da20 con 0x7ff1c80fad90 2024-02-19T22:08:58.779 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:58.776+0000 7ff1cc96e700 1 -- 172.21.15.52:0/971498184 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7ff1c4010470 con 0x7ff1c80fad90 2024-02-19T22:08:58.779 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:58.776+0000 7ff1cc96e700 1 -- 172.21.15.52:0/971498184 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1c400f5b0 con 0x7ff1c80fad90 2024-02-19T22:08:58.779 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:58.777+0000 7ff1cfbd4700 1 -- 172.21.15.52:0/971498184 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff1c80fad90 msgr2=0x7ff1c80fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:58.779 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:58.777+0000 7ff1cfbd4700 1 --2- 172.21.15.52:0/971498184 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff1c80fad90 0x7ff1c80fb1b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7ff1c400b870 tx=0x7ff1c4006f70).stop 2024-02-19T22:08:58.780 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:58.777+0000 7ff1cfbd4700 1 -- 172.21.15.52:0/971498184 shutdown_connections 2024-02-19T22:08:58.780 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:58.777+0000 7ff1cfbd4700 1 --2- 172.21.15.52:0/971498184 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff1c80fad90 0x7ff1c80fb1b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:58.780 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:58.777+0000 7ff1cfbd4700 1 -- 172.21.15.52:0/971498184 >> 172.21.15.52:0/971498184 conn(0x7ff1c80f6530 msgr2=0x7ff1c80f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:58.780 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:58.777+0000 7ff1cfbd4700 1 -- 172.21.15.52:0/971498184 shutdown_connections 2024-02-19T22:08:58.780 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:58.777+0000 7ff1cfbd4700 1 -- 172.21.15.52:0/971498184 wait complete. 2024-02-19T22:08:58.780 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:58.778+0000 7ff1cfbd4700 1 Processor -- start 2024-02-19T22:08:58.782 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:58.778+0000 7ff1cfbd4700 1 -- start start 2024-02-19T22:08:58.782 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:58.778+0000 7ff1cfbd4700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff1c80fad90 0x7ff1c8108920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:58.782 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:58.778+0000 7ff1cfbd4700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c8108e10 con 0x7ff1c80fad90 2024-02-19T22:08:58.782 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:58.778+0000 7ff1cd970700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff1c80fad90 0x7ff1c8108920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:58.782 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:58.778+0000 7ff1cd970700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff1c80fad90 0x7ff1c8108920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51438/0 (socket says 172.21.15.52:51438) 2024-02-19T22:08:58.782 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:58.778+0000 7ff1cd970700 1 -- 172.21.15.52:0/2193764349 learned_addr learned my addr 172.21.15.52:0/2193764349 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:08:58.782 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:58.778+0000 7ff1cd970700 1 -- 172.21.15.52:0/2193764349 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c400b480 con 0x7ff1c80fad90 2024-02-19T22:08:58.783 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:58.779+0000 7ff1cd970700 1 --2- 172.21.15.52:0/2193764349 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff1c80fad90 0x7ff1c8108920 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7ff1c4000c00 tx=0x7ff1c4006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:58.783 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:58.779+0000 7ff1b67fc700 1 -- 172.21.15.52:0/2193764349 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1c40109e0 con 0x7ff1c80fad90 2024-02-19T22:08:58.783 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:58.779+0000 7ff1cfbd4700 1 -- 172.21.15.52:0/2193764349 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1c8109010 con 0x7ff1c80fad90 2024-02-19T22:08:58.783 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:58.779+0000 7ff1b67fc700 1 -- 172.21.15.52:0/2193764349 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7ff1c400fcb0 con 0x7ff1c80fad90 2024-02-19T22:08:58.783 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:58.779+0000 7ff1b67fc700 1 -- 172.21.15.52:0/2193764349 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1c4019cc0 con 0x7ff1c80fad90 2024-02-19T22:08:58.783 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:58.779+0000 7ff1cfbd4700 1 -- 172.21.15.52:0/2193764349 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1c81094a0 con 0x7ff1c80fad90 2024-02-19T22:08:58.783 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:58.780+0000 7ff1b67fc700 1 -- 172.21.15.52:0/2193764349 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7ff1c4016030 con 0x7ff1c80fad90 2024-02-19T22:08:58.783 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:58.780+0000 7ff1b67fc700 1 --2- 172.21.15.52:0/2193764349 >> [v2:172.21.15.52:6800/384714397,v1:172.21.15.52:6801/384714397] conn(0x7ff1b002fe10 0x7ff1b00322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:58.783 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:58.780+0000 7ff1b67fc700 1 -- 172.21.15.52:0/2193764349 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7ff1c4044140 con 0x7ff1c80fad90 2024-02-19T22:08:58.783 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:58.780+0000 7ff1cd16f700 1 -- 172.21.15.52:0/2193764349 >> [v2:172.21.15.52:6800/384714397,v1:172.21.15.52:6801/384714397] conn(0x7ff1b002fe10 msgr2=0x7ff1b00322d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/384714397 2024-02-19T22:08:58.783 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:58.780+0000 7ff1cd16f700 1 --2- 172.21.15.52:0/2193764349 >> [v2:172.21.15.52:6800/384714397,v1:172.21.15.52:6801/384714397] conn(0x7ff1b002fe10 0x7ff1b00322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-19T22:08:58.783 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:58.780+0000 7ff1cfbd4700 1 -- 172.21.15.52:0/2193764349 --> [v2:172.21.15.52:6800/384714397,v1:172.21.15.52:6801/384714397] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff1c8059100 con 0x7ff1b002fe10 2024-02-19T22:08:58.982 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:58.981+0000 7ff1cd16f700 1 -- 172.21.15.52:0/2193764349 >> [v2:172.21.15.52:6800/384714397,v1:172.21.15.52:6801/384714397] conn(0x7ff1b002fe10 msgr2=0x7ff1b00322d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/384714397 2024-02-19T22:08:58.982 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:58.981+0000 7ff1cd16f700 1 --2- 172.21.15.52:0/2193764349 >> [v2:172.21.15.52:6800/384714397,v1:172.21.15.52:6801/384714397] conn(0x7ff1b002fe10 0x7ff1b00322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-19T22:08:59.383 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:59.382+0000 7ff1cd16f700 1 -- 172.21.15.52:0/2193764349 >> [v2:172.21.15.52:6800/384714397,v1:172.21.15.52:6801/384714397] conn(0x7ff1b002fe10 msgr2=0x7ff1b00322d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/384714397 2024-02-19T22:08:59.383 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:08:59.382+0000 7ff1cd16f700 1 --2- 172.21.15.52:0/2193764349 >> [v2:172.21.15.52:6800/384714397,v1:172.21.15.52:6801/384714397] conn(0x7ff1b002fe10 0x7ff1b00322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-19T22:09:00.184 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:00.183+0000 7ff1cd16f700 1 -- 172.21.15.52:0/2193764349 >> [v2:172.21.15.52:6800/384714397,v1:172.21.15.52:6801/384714397] conn(0x7ff1b002fe10 msgr2=0x7ff1b00322d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/384714397 2024-02-19T22:09:00.184 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:00.183+0000 7ff1cd16f700 1 --2- 172.21.15.52:0/2193764349 >> [v2:172.21.15.52:6800/384714397,v1:172.21.15.52:6801/384714397] conn(0x7ff1b002fe10 0x7ff1b00322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-02-19T22:09:01.056 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:01.054+0000 7ff1b67fc700 1 -- 172.21.15.52:0/2193764349 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mgrmap(e 6) v1 ==== 35970+0+0 (secure 0 0 0) 0x7ff1c402ece0 con 0x7ff1c80fad90 2024-02-19T22:09:01.056 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:01.055+0000 7ff1b67fc700 1 -- 172.21.15.52:0/2193764349 >> [v2:172.21.15.52:6800/384714397,v1:172.21.15.52:6801/384714397] conn(0x7ff1b002fe10 msgr2=0x7ff1b00322d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T22:09:01.056 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:01.055+0000 7ff1b67fc700 1 --2- 172.21.15.52:0/2193764349 >> [v2:172.21.15.52:6800/384714397,v1:172.21.15.52:6801/384714397] conn(0x7ff1b002fe10 0x7ff1b00322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T22:09:02.057 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:02.056+0000 7ff1b67fc700 1 -- 172.21.15.52:0/2193764349 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7ff1c40194e0 con 0x7ff1c80fad90 2024-02-19T22:09:02.058 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:02.056+0000 7ff1b67fc700 1 --2- 172.21.15.52:0/2193764349 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7ff1b0033780 0x7ff1b0035b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:02.058 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:02.056+0000 7ff1b67fc700 1 -- 172.21.15.52:0/2193764349 --> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff1c8059100 con 0x7ff1b0033780 2024-02-19T22:09:02.058 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:02.057+0000 7ff1cd16f700 1 --2- 172.21.15.52:0/2193764349 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7ff1b0033780 0x7ff1b0035b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:02.058 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:02.057+0000 7ff1cd16f700 1 --2- 172.21.15.52:0/2193764349 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7ff1b0033780 0x7ff1b0035b70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7ff1b80036d0 tx=0x7ff1b8006ca0).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:02.059 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:02.058+0000 7ff1b67fc700 1 -- 172.21.15.52:0/2193764349 <== mgr.14114 v2:172.21.15.52:6800/892530721 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7ff1c8059100 con 0x7ff1b0033780 2024-02-19T22:09:02.066 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:02.064+0000 7ff1cfbd4700 1 -- 172.21.15.52:0/2193764349 --> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7ff1c8045ad0 con 0x7ff1b0033780 2024-02-19T22:09:02.066 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:02.065+0000 7ff1b67fc700 1 -- 172.21.15.52:0/2193764349 <== mgr.14114 v2:172.21.15.52:6800/892530721 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7ff1c8045ad0 con 0x7ff1b0033780 2024-02-19T22:09:02.066 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: { 2024-02-19T22:09:02.067 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-02-19T22:09:02.068 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "initialized": true 2024-02-19T22:09:02.068 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: } 2024-02-19T22:09:02.068 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:02.065+0000 7ff1cfbd4700 1 -- 172.21.15.52:0/2193764349 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7ff1b0033780 msgr2=0x7ff1b0035b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:02.068 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:02.065+0000 7ff1cfbd4700 1 --2- 172.21.15.52:0/2193764349 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7ff1b0033780 0x7ff1b0035b70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7ff1b80036d0 tx=0x7ff1b8006ca0).stop 2024-02-19T22:09:02.068 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:02.065+0000 7ff1cfbd4700 1 -- 172.21.15.52:0/2193764349 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff1c80fad90 msgr2=0x7ff1c8108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:02.068 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:02.065+0000 7ff1cfbd4700 1 --2- 172.21.15.52:0/2193764349 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff1c80fad90 0x7ff1c8108920 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7ff1c4000c00 tx=0x7ff1c4006f70).stop 2024-02-19T22:09:02.068 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:02.065+0000 7ff1cfbd4700 1 -- 172.21.15.52:0/2193764349 shutdown_connections 2024-02-19T22:09:02.068 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:02.065+0000 7ff1cfbd4700 1 --2- 172.21.15.52:0/2193764349 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7ff1b0033780 0x7ff1b0035b70 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:02.068 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:02.065+0000 7ff1cfbd4700 1 --2- 172.21.15.52:0/2193764349 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff1c80fad90 0x7ff1c8108920 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:02.068 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:02.065+0000 7ff1cfbd4700 1 --2- 172.21.15.52:0/2193764349 >> [v2:172.21.15.52:6800/384714397,v1:172.21.15.52:6801/384714397] conn(0x7ff1b002fe10 0x7ff1b00322d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T22:09:02.068 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:02.065+0000 7ff1cfbd4700 1 -- 172.21.15.52:0/2193764349 >> 172.21.15.52:0/2193764349 conn(0x7ff1c80f6530 msgr2=0x7ff1c80ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:02.068 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:02.065+0000 7ff1cfbd4700 1 -- 172.21.15.52:0/2193764349 shutdown_connections 2024-02-19T22:09:02.068 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:02.065+0000 7ff1cfbd4700 1 -- 172.21.15.52:0/2193764349 wait complete. 2024-02-19T22:09:02.343 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:02 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:09:01.003989+0000 mon.smithi052 (mon.0) 29 : cluster [INF] Active manager daemon smithi052.ghqrtk restarted 2024-02-19T22:09:02.343 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:02 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:09:01.004354+0000 mon.smithi052 (mon.0) 30 : cluster [INF] Activating manager daemon smithi052.ghqrtk 2024-02-19T22:09:02.343 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:02 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:09:01.006268+0000 mon.smithi052 (mon.0) 31 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-02-19T22:09:02.344 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:02 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:09:01.056861+0000 mon.smithi052 (mon.0) 32 : cluster [DBG] mgrmap e6: smithi052.ghqrtk(active, starting, since 0.0525787s) 2024-02-19T22:09:02.344 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:02 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:01.057930+0000 mon.smithi052 (mon.0) 33 : audit [DBG] from='mgr.14114 172.21.15.52:0/3613687413' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix": "mon metadata", "id": "smithi052"}]: dispatch 2024-02-19T22:09:02.344 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:02 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:01.058490+0000 mon.smithi052 (mon.0) 34 : audit [DBG] from='mgr.14114 172.21.15.52:0/3613687413' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix": "mgr metadata", "who": "smithi052.ghqrtk", "id": "smithi052.ghqrtk"}]: dispatch 2024-02-19T22:09:02.344 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:02 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:01.059176+0000 mon.smithi052 (mon.0) 35 : audit [DBG] from='mgr.14114 172.21.15.52:0/3613687413' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T22:09:02.344 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:02 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:01.059418+0000 mon.smithi052 (mon.0) 36 : audit [DBG] from='mgr.14114 172.21.15.52:0/3613687413' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T22:09:02.344 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:02 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:01.059625+0000 mon.smithi052 (mon.0) 37 : audit [DBG] from='mgr.14114 172.21.15.52:0/3613687413' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T22:09:02.344 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:02 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:09:01.073556+0000 mon.smithi052 (mon.0) 38 : cluster [INF] Manager daemon smithi052.ghqrtk is now available 2024-02-19T22:09:02.344 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:02 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:01.087117+0000 mon.smithi052 (mon.0) 39 : audit [INF] from='mgr.14114 172.21.15.52:0/3613687413' entity='mgr.smithi052.ghqrtk' 2024-02-19T22:09:02.344 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:02 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:01.089766+0000 mon.smithi052 (mon.0) 40 : audit [INF] from='mgr.14114 172.21.15.52:0/3613687413' entity='mgr.smithi052.ghqrtk' 2024-02-19T22:09:02.344 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:02 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:01.092372+0000 mon.smithi052 (mon.0) 41 : audit [INF] from='mgr.14114 172.21.15.52:0/3613687413' entity='mgr.smithi052.ghqrtk' 2024-02-19T22:09:02.344 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:02 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:01.186941+0000 mon.smithi052 (mon.0) 42 : audit [INF] from='mgr.14114 172.21.15.52:0/3613687413' entity='mgr.smithi052.ghqrtk' 2024-02-19T22:09:02.344 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:02 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:01.187670+0000 mon.smithi052 (mon.0) 43 : audit [DBG] from='mgr.14114 172.21.15.52:0/3613687413' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T22:09:02.344 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:02 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:01.189439+0000 mon.smithi052 (mon.0) 44 : audit [DBG] from='mgr.14114 172.21.15.52:0/3613687413' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T22:09:02.344 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:02 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:01.190898+0000 mon.smithi052 (mon.0) 45 : audit [DBG] from='mgr.14114 172.21.15.52:0/3613687413' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T22:09:02.345 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:02 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:01.201201+0000 mon.smithi052 (mon.0) 46 : audit [INF] from='mgr.14114 172.21.15.52:0/3613687413' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi052.ghqrtk/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:09:02.345 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:02 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:01.202387+0000 mon.smithi052 (mon.0) 47 : audit [INF] from='mgr.14114 172.21.15.52:0/3613687413' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi052.ghqrtk/trash_purge_schedule"}]: dispatch 2024-02-19T22:09:02.645 INFO:teuthology.orchestra.run.smithi052.stderr:mgr epoch 5 is available 2024-02-19T22:09:02.646 INFO:teuthology.orchestra.run.smithi052.stderr:Setting orchestrator backend to cephadm... 2024-02-19T22:09:02.646 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi052 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphvym4zi3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp6ou7w4ul:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2024-02-19T22:09:03.226 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.220+0000 7f5223708700 1 Processor -- start 2024-02-19T22:09:03.226 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.220+0000 7f5223708700 1 -- start start 2024-02-19T22:09:03.226 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.220+0000 7f5223708700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f521c0703b0 0x7f521c100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:03.226 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.220+0000 7f5223708700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f521c0fc8c0 con 0x7f521c0703b0 2024-02-19T22:09:03.226 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.221+0000 7f52214a4700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f521c0703b0 0x7f521c100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:03.226 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.221+0000 7f52214a4700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f521c0703b0 0x7f521c100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51482/0 (socket says 172.21.15.52:51482) 2024-02-19T22:09:03.226 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.221+0000 7f52214a4700 1 -- 172.21.15.52:0/468009998 learned_addr learned my addr 172.21.15.52:0/468009998 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:09:03.226 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.221+0000 7f52214a4700 1 -- 172.21.15.52:0/468009998 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f521c1013c0 con 0x7f521c0703b0 2024-02-19T22:09:03.227 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.221+0000 7f52214a4700 1 --2- 172.21.15.52:0/468009998 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f521c0703b0 0x7f521c100ed0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f520c00b5f0 tx=0x7f520c006ee0).ready entity=mon.0 client_cookie=b7be9a77a626d85b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:03.227 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.221+0000 7f520bfff700 1 -- 172.21.15.52:0/468009998 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f520c00d600 con 0x7f521c0703b0 2024-02-19T22:09:03.227 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.221+0000 7f520bfff700 1 -- 172.21.15.52:0/468009998 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f520c00dbf0 con 0x7f521c0703b0 2024-02-19T22:09:03.227 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.221+0000 7f520bfff700 1 -- 172.21.15.52:0/468009998 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f520c011cb0 con 0x7f521c0703b0 2024-02-19T22:09:03.227 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.222+0000 7f5223708700 1 -- 172.21.15.52:0/468009998 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f521c0703b0 msgr2=0x7f521c100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:03.227 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.222+0000 7f5223708700 1 --2- 172.21.15.52:0/468009998 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f521c0703b0 0x7f521c100ed0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f520c00b5f0 tx=0x7f520c006ee0).stop 2024-02-19T22:09:03.227 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.222+0000 7f5223708700 1 -- 172.21.15.52:0/468009998 shutdown_connections 2024-02-19T22:09:03.227 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.222+0000 7f5223708700 1 --2- 172.21.15.52:0/468009998 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f521c0703b0 0x7f521c100ed0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:03.227 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.222+0000 7f5223708700 1 -- 172.21.15.52:0/468009998 >> 172.21.15.52:0/468009998 conn(0x7f521c0f6530 msgr2=0x7f521c0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:03.227 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.222+0000 7f5223708700 1 -- 172.21.15.52:0/468009998 shutdown_connections 2024-02-19T22:09:03.227 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.222+0000 7f5223708700 1 -- 172.21.15.52:0/468009998 wait complete. 2024-02-19T22:09:03.227 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.223+0000 7f5223708700 1 Processor -- start 2024-02-19T22:09:03.228 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.223+0000 7f5223708700 1 -- start start 2024-02-19T22:09:03.228 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.223+0000 7f5223708700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f521c0703b0 0x7f521c10cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:03.228 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.223+0000 7f5223708700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f521c10d2c0 con 0x7f521c0703b0 2024-02-19T22:09:03.228 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.223+0000 7f52214a4700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f521c0703b0 0x7f521c10cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:03.228 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.223+0000 7f52214a4700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f521c0703b0 0x7f521c10cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51490/0 (socket says 172.21.15.52:51490) 2024-02-19T22:09:03.228 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.223+0000 7f52214a4700 1 -- 172.21.15.52:0/2258935174 learned_addr learned my addr 172.21.15.52:0/2258935174 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:09:03.228 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.224+0000 7f52214a4700 1 -- 172.21.15.52:0/2258935174 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f520c00b200 con 0x7f521c0703b0 2024-02-19T22:09:03.228 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.224+0000 7f52214a4700 1 --2- 172.21.15.52:0/2258935174 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f521c0703b0 0x7f521c10cdd0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f520c00df90 tx=0x7f520c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:03.228 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.224+0000 7f5209ffb700 1 -- 172.21.15.52:0/2258935174 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f520c01b410 con 0x7f521c0703b0 2024-02-19T22:09:03.228 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.224+0000 7f5209ffb700 1 -- 172.21.15.52:0/2258935174 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f520c01ba00 con 0x7f521c0703b0 2024-02-19T22:09:03.228 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.224+0000 7f5223708700 1 -- 172.21.15.52:0/2258935174 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f521c10d4c0 con 0x7f521c0703b0 2024-02-19T22:09:03.228 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.224+0000 7f5209ffb700 1 -- 172.21.15.52:0/2258935174 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f520c01aa30 con 0x7f521c0703b0 2024-02-19T22:09:03.228 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.224+0000 7f5223708700 1 -- 172.21.15.52:0/2258935174 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f521c10e100 con 0x7f521c0703b0 2024-02-19T22:09:03.229 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.225+0000 7f5223708700 1 -- 172.21.15.52:0/2258935174 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f521c10d950 con 0x7f521c0703b0 2024-02-19T22:09:03.230 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.225+0000 7f5209ffb700 1 -- 172.21.15.52:0/2258935174 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f520c00f070 con 0x7f521c0703b0 2024-02-19T22:09:03.230 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.226+0000 7f5209ffb700 1 --2- 172.21.15.52:0/2258935174 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7f520402fe70 0x7f5204032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:03.230 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.226+0000 7f5209ffb700 1 -- 172.21.15.52:0/2258935174 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f520c041880 con 0x7f521c0703b0 2024-02-19T22:09:03.230 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.226+0000 7f5220ca3700 1 --2- 172.21.15.52:0/2258935174 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7f520402fe70 0x7f5204032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:03.230 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.226+0000 7f5220ca3700 1 --2- 172.21.15.52:0/2258935174 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7f520402fe70 0x7f5204032330 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f52180096a0 tx=0x7f5218006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:03.230 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.230+0000 7f5209ffb700 1 -- 172.21.15.52:0/2258935174 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f520c023b50 con 0x7f521c0703b0 2024-02-19T22:09:03.342 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:03 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:02.059194+0000 mgr.smithi052.ghqrtk (mgr.14114) 1 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-19T22:09:03.342 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:03 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:09:02.060444+0000 mon.smithi052 (mon.0) 48 : cluster [DBG] mgrmap e7: smithi052.ghqrtk(active, since 1.05616s) 2024-02-19T22:09:03.342 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:03 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:02.066264+0000 mgr.smithi052.ghqrtk (mgr.14114) 2 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-19T22:09:03.420 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.419+0000 7f5223708700 1 -- 172.21.15.52:0/2258935174 --> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f521c10e400 con 0x7f520402fe70 2024-02-19T22:09:03.425 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.424+0000 7f5209ffb700 1 -- 172.21.15.52:0/2258935174 <== mgr.14114 v2:172.21.15.52:6800/892530721 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f521c10e400 con 0x7f520402fe70 2024-02-19T22:09:03.429 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.426+0000 7f5223708700 1 -- 172.21.15.52:0/2258935174 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7f520402fe70 msgr2=0x7f5204032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:03.429 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.426+0000 7f5223708700 1 --2- 172.21.15.52:0/2258935174 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7f520402fe70 0x7f5204032330 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f52180096a0 tx=0x7f5218006b40).stop 2024-02-19T22:09:03.429 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.426+0000 7f5223708700 1 -- 172.21.15.52:0/2258935174 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f521c0703b0 msgr2=0x7f521c10cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:03.429 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.426+0000 7f5223708700 1 --2- 172.21.15.52:0/2258935174 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f521c0703b0 0x7f521c10cdd0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f520c00df90 tx=0x7f520c006ee0).stop 2024-02-19T22:09:03.429 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.427+0000 7f5223708700 1 -- 172.21.15.52:0/2258935174 shutdown_connections 2024-02-19T22:09:03.429 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.427+0000 7f5223708700 1 --2- 172.21.15.52:0/2258935174 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7f520402fe70 0x7f5204032330 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:03.429 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.427+0000 7f5223708700 1 --2- 172.21.15.52:0/2258935174 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f521c0703b0 0x7f521c10cdd0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:03.429 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.427+0000 7f5223708700 1 -- 172.21.15.52:0/2258935174 >> 172.21.15.52:0/2258935174 conn(0x7f521c0f6530 msgr2=0x7f521c0f7f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:03.429 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.427+0000 7f5223708700 1 -- 172.21.15.52:0/2258935174 shutdown_connections 2024-02-19T22:09:03.429 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:03.427+0000 7f5223708700 1 -- 172.21.15.52:0/2258935174 wait complete. 2024-02-19T22:09:04.027 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi052 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphvym4zi3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp6ou7w4ul:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2024-02-19T22:09:04.307 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:04 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:09:03.061317+0000 mon.smithi052 (mon.0) 49 : cluster [DBG] mgrmap e8: smithi052.ghqrtk(active, since 2s) 2024-02-19T22:09:04.307 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:04 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:03.421749+0000 mgr.smithi052.ghqrtk (mgr.14114) 3 : audit [DBG] from='client.14124 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:09:04.307 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:04 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:03.425679+0000 mon.smithi052 (mon.0) 50 : audit [INF] from='mgr.14114 172.21.15.52:0/3613687413' entity='mgr.smithi052.ghqrtk' 2024-02-19T22:09:04.307 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:04 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:03.428801+0000 mon.smithi052 (mon.0) 51 : audit [DBG] from='mgr.14114 172.21.15.52:0/3613687413' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T22:09:04.568 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.566+0000 7fde13ad8700 1 Processor -- start 2024-02-19T22:09:04.568 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.566+0000 7fde13ad8700 1 -- start start 2024-02-19T22:09:04.568 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.566+0000 7fde13ad8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fde0c0fbea0 0x7fde0c0fc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:04.568 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.566+0000 7fde13ad8700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde0c0fc7b0 con 0x7fde0c0fbea0 2024-02-19T22:09:04.569 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.567+0000 7fde11874700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fde0c0fbea0 0x7fde0c0fc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:04.569 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.567+0000 7fde11874700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fde0c0fbea0 0x7fde0c0fc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:55762/0 (socket says 172.21.15.52:55762) 2024-02-19T22:09:04.569 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.567+0000 7fde11874700 1 -- 172.21.15.52:0/3090721331 learned_addr learned my addr 172.21.15.52:0/3090721331 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:09:04.569 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.567+0000 7fde11874700 1 -- 172.21.15.52:0/3090721331 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde0c0fc8f0 con 0x7fde0c0fbea0 2024-02-19T22:09:04.569 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.567+0000 7fde11874700 1 --2- 172.21.15.52:0/3090721331 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fde0c0fbea0 0x7fde0c0fc2c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fddfc00b5f0 tx=0x7fddfc006ee0).ready entity=mon.0 client_cookie=1dd1d4e17e974dec server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:04.571 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.567+0000 7fde10872700 1 -- 172.21.15.52:0/3090721331 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fddfc00d600 con 0x7fde0c0fbea0 2024-02-19T22:09:04.571 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.567+0000 7fde10872700 1 -- 172.21.15.52:0/3090721331 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fddfc00dbf0 con 0x7fde0c0fbea0 2024-02-19T22:09:04.571 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.567+0000 7fde10872700 1 -- 172.21.15.52:0/3090721331 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fddfc011cb0 con 0x7fde0c0fbea0 2024-02-19T22:09:04.571 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.568+0000 7fde13ad8700 1 -- 172.21.15.52:0/3090721331 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fde0c0fbea0 msgr2=0x7fde0c0fc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:04.571 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.568+0000 7fde13ad8700 1 --2- 172.21.15.52:0/3090721331 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fde0c0fbea0 0x7fde0c0fc2c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fddfc00b5f0 tx=0x7fddfc006ee0).stop 2024-02-19T22:09:04.571 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.568+0000 7fde13ad8700 1 -- 172.21.15.52:0/3090721331 shutdown_connections 2024-02-19T22:09:04.571 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.568+0000 7fde13ad8700 1 --2- 172.21.15.52:0/3090721331 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fde0c0fbea0 0x7fde0c0fc2c0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:04.571 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.568+0000 7fde13ad8700 1 -- 172.21.15.52:0/3090721331 >> 172.21.15.52:0/3090721331 conn(0x7fde0c0f76a0 msgr2=0x7fde0c0f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:04.571 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.568+0000 7fde13ad8700 1 -- 172.21.15.52:0/3090721331 shutdown_connections 2024-02-19T22:09:04.571 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.568+0000 7fde13ad8700 1 -- 172.21.15.52:0/3090721331 wait complete. 2024-02-19T22:09:04.571 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.569+0000 7fde13ad8700 1 Processor -- start 2024-02-19T22:09:04.571 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.569+0000 7fde13ad8700 1 -- start start 2024-02-19T22:09:04.571 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.569+0000 7fde13ad8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fde0c0fbea0 0x7fde0c10ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:04.571 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.569+0000 7fde13ad8700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde0c10b080 con 0x7fde0c0fbea0 2024-02-19T22:09:04.572 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.569+0000 7fde11874700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fde0c0fbea0 0x7fde0c10ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:04.572 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.569+0000 7fde11874700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fde0c0fbea0 0x7fde0c10ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:55778/0 (socket says 172.21.15.52:55778) 2024-02-19T22:09:04.572 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.569+0000 7fde11874700 1 -- 172.21.15.52:0/220966729 learned_addr learned my addr 172.21.15.52:0/220966729 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:09:04.573 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.569+0000 7fde11874700 1 -- 172.21.15.52:0/220966729 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddfc00b200 con 0x7fde0c0fbea0 2024-02-19T22:09:04.573 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.569+0000 7fde11874700 1 --2- 172.21.15.52:0/220966729 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fde0c0fbea0 0x7fde0c10ab90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fddfc00bf40 tx=0x7fddfc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:04.573 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.570+0000 7fddfa7fc700 1 -- 172.21.15.52:0/220966729 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fddfc01b570 con 0x7fde0c0fbea0 2024-02-19T22:09:04.573 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.570+0000 7fddfa7fc700 1 -- 172.21.15.52:0/220966729 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fddfc01bb60 con 0x7fde0c0fbea0 2024-02-19T22:09:04.573 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.570+0000 7fde13ad8700 1 -- 172.21.15.52:0/220966729 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde0c10b280 con 0x7fde0c0fbea0 2024-02-19T22:09:04.573 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.570+0000 7fddfa7fc700 1 -- 172.21.15.52:0/220966729 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fddfc01aa20 con 0x7fde0c0fbea0 2024-02-19T22:09:04.573 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.570+0000 7fde13ad8700 1 -- 172.21.15.52:0/220966729 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde0c10b710 con 0x7fde0c0fbea0 2024-02-19T22:09:04.573 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.570+0000 7fddfa7fc700 1 -- 172.21.15.52:0/220966729 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fddfc00f070 con 0x7fde0c0fbea0 2024-02-19T22:09:04.573 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.571+0000 7fddfa7fc700 1 --2- 172.21.15.52:0/220966729 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7fddf402fe70 0x7fddf4032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:04.573 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.571+0000 7fddfa7fc700 1 -- 172.21.15.52:0/220966729 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fddfc0414e0 con 0x7fde0c0fbea0 2024-02-19T22:09:04.573 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.571+0000 7fde13ad8700 1 -- 172.21.15.52:0/220966729 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde0c045ad0 con 0x7fde0c0fbea0 2024-02-19T22:09:04.573 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.571+0000 7fde11073700 1 --2- 172.21.15.52:0/220966729 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7fddf402fe70 0x7fddf4032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:04.573 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.571+0000 7fde11073700 1 --2- 172.21.15.52:0/220966729 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7fddf402fe70 0x7fddf4032330 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fde080096a0 tx=0x7fde08006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:04.576 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.575+0000 7fddfa7fc700 1 -- 172.21.15.52:0/220966729 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fddfc016070 con 0x7fde0c0fbea0 2024-02-19T22:09:04.736 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.735+0000 7fde13ad8700 1 -- 172.21.15.52:0/220966729 --> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fde0c10c030 con 0x7fddf402fe70 2024-02-19T22:09:04.736 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.735+0000 7fddfa7fc700 1 -- 172.21.15.52:0/220966729 <== mgr.14114 v2:172.21.15.52:6800/892530721 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fde0c10c030 con 0x7fddf402fe70 2024-02-19T22:09:04.737 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: value unchanged 2024-02-19T22:09:04.739 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.737+0000 7fde13ad8700 1 -- 172.21.15.52:0/220966729 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7fddf402fe70 msgr2=0x7fddf4032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:04.739 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.737+0000 7fde13ad8700 1 --2- 172.21.15.52:0/220966729 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7fddf402fe70 0x7fddf4032330 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fde080096a0 tx=0x7fde08006b40).stop 2024-02-19T22:09:04.739 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.737+0000 7fde13ad8700 1 -- 172.21.15.52:0/220966729 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fde0c0fbea0 msgr2=0x7fde0c10ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:04.739 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.737+0000 7fde13ad8700 1 --2- 172.21.15.52:0/220966729 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fde0c0fbea0 0x7fde0c10ab90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fddfc00bf40 tx=0x7fddfc006ee0).stop 2024-02-19T22:09:04.739 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.737+0000 7fde13ad8700 1 -- 172.21.15.52:0/220966729 shutdown_connections 2024-02-19T22:09:04.739 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.737+0000 7fde13ad8700 1 --2- 172.21.15.52:0/220966729 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7fddf402fe70 0x7fddf4032330 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:04.739 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.737+0000 7fde13ad8700 1 --2- 172.21.15.52:0/220966729 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fde0c0fbea0 0x7fde0c10ab90 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:04.739 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.737+0000 7fde13ad8700 1 -- 172.21.15.52:0/220966729 >> 172.21.15.52:0/220966729 conn(0x7fde0c0f76a0 msgr2=0x7fde0c0f8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:04.739 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.737+0000 7fde13ad8700 1 -- 172.21.15.52:0/220966729 shutdown_connections 2024-02-19T22:09:04.739 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:04.737+0000 7fde13ad8700 1 -- 172.21.15.52:0/220966729 wait complete. 2024-02-19T22:09:05.341 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:05 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:04.736853+0000 mgr.smithi052.ghqrtk (mgr.14114) 4 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:09:05.400 INFO:teuthology.orchestra.run.smithi052.stderr:Generating ssh key... 2024-02-19T22:09:05.400 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi052 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphvym4zi3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp6ou7w4ul:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2024-02-19T22:09:05.943 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:05.939+0000 7ff6ffdb1700 1 Processor -- start 2024-02-19T22:09:05.943 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:05.939+0000 7ff6ffdb1700 1 -- start start 2024-02-19T22:09:05.943 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:05.939+0000 7ff6ffdb1700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff6f80fbe40 0x7ff6f80fc260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:05.943 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:05.939+0000 7ff6ffdb1700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f80fc750 con 0x7ff6f80fbe40 2024-02-19T22:09:05.943 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:05.940+0000 7ff6fdb4d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff6f80fbe40 0x7ff6f80fc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:05.943 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:05.940+0000 7ff6fdb4d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff6f80fbe40 0x7ff6f80fc260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:55786/0 (socket says 172.21.15.52:55786) 2024-02-19T22:09:05.943 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:05.940+0000 7ff6fdb4d700 1 -- 172.21.15.52:0/2915713313 learned_addr learned my addr 172.21.15.52:0/2915713313 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:09:05.943 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:05.940+0000 7ff6fdb4d700 1 -- 172.21.15.52:0/2915713313 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f80fc890 con 0x7ff6f80fbe40 2024-02-19T22:09:05.943 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:05.940+0000 7ff6fdb4d700 1 --2- 172.21.15.52:0/2915713313 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff6f80fbe40 0x7ff6f80fc260 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7ff6f400b5f0 tx=0x7ff6f4006ee0).ready entity=mon.0 client_cookie=2827e8ca07a954e1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:05.944 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:05.940+0000 7ff6fcb4b700 1 -- 172.21.15.52:0/2915713313 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6f400d600 con 0x7ff6f80fbe40 2024-02-19T22:09:05.944 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:05.940+0000 7ff6fcb4b700 1 -- 172.21.15.52:0/2915713313 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7ff6f400dbf0 con 0x7ff6f80fbe40 2024-02-19T22:09:05.944 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:05.940+0000 7ff6fcb4b700 1 -- 172.21.15.52:0/2915713313 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6f4011cb0 con 0x7ff6f80fbe40 2024-02-19T22:09:05.944 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:05.941+0000 7ff6ffdb1700 1 -- 172.21.15.52:0/2915713313 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff6f80fbe40 msgr2=0x7ff6f80fc260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:05.944 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:05.941+0000 7ff6ffdb1700 1 --2- 172.21.15.52:0/2915713313 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff6f80fbe40 0x7ff6f80fc260 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7ff6f400b5f0 tx=0x7ff6f4006ee0).stop 2024-02-19T22:09:05.944 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:05.941+0000 7ff6ffdb1700 1 -- 172.21.15.52:0/2915713313 shutdown_connections 2024-02-19T22:09:05.944 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:05.941+0000 7ff6ffdb1700 1 --2- 172.21.15.52:0/2915713313 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff6f80fbe40 0x7ff6f80fc260 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:05.944 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:05.941+0000 7ff6ffdb1700 1 -- 172.21.15.52:0/2915713313 >> 172.21.15.52:0/2915713313 conn(0x7ff6f80f7640 msgr2=0x7ff6f80f9a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:05.944 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:05.941+0000 7ff6ffdb1700 1 -- 172.21.15.52:0/2915713313 shutdown_connections 2024-02-19T22:09:05.944 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:05.941+0000 7ff6ffdb1700 1 -- 172.21.15.52:0/2915713313 wait complete. 2024-02-19T22:09:05.946 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:05.942+0000 7ff6ffdb1700 1 Processor -- start 2024-02-19T22:09:05.946 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:05.942+0000 7ff6ffdb1700 1 -- start start 2024-02-19T22:09:05.946 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:05.942+0000 7ff6ffdb1700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff6f80fbe40 0x7ff6f810aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:05.946 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:05.942+0000 7ff6ffdb1700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f810b090 con 0x7ff6f80fbe40 2024-02-19T22:09:05.946 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:05.942+0000 7ff6fdb4d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff6f80fbe40 0x7ff6f810aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:05.946 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:05.942+0000 7ff6fdb4d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff6f80fbe40 0x7ff6f810aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:55800/0 (socket says 172.21.15.52:55800) 2024-02-19T22:09:05.946 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:05.942+0000 7ff6fdb4d700 1 -- 172.21.15.52:0/1268637922 learned_addr learned my addr 172.21.15.52:0/1268637922 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:09:05.946 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:05.943+0000 7ff6fdb4d700 1 -- 172.21.15.52:0/1268637922 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f400b200 con 0x7ff6f80fbe40 2024-02-19T22:09:05.946 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:05.943+0000 7ff6fdb4d700 1 --2- 172.21.15.52:0/1268637922 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff6f80fbe40 0x7ff6f810aba0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7ff6f4002f40 tx=0x7ff6f4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:05.946 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:05.943+0000 7ff6ea7fc700 1 -- 172.21.15.52:0/1268637922 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6f401b410 con 0x7ff6f80fbe40 2024-02-19T22:09:05.946 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:05.943+0000 7ff6ea7fc700 1 -- 172.21.15.52:0/1268637922 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7ff6f401ba00 con 0x7ff6f80fbe40 2024-02-19T22:09:05.946 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:05.943+0000 7ff6ffdb1700 1 -- 172.21.15.52:0/1268637922 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6f810b290 con 0x7ff6f80fbe40 2024-02-19T22:09:05.946 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:05.943+0000 7ff6ea7fc700 1 -- 172.21.15.52:0/1268637922 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6f401a980 con 0x7ff6f80fbe40 2024-02-19T22:09:05.947 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:05.943+0000 7ff6ffdb1700 1 -- 172.21.15.52:0/1268637922 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6f810b720 con 0x7ff6f80fbe40 2024-02-19T22:09:05.947 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:05.944+0000 7ff6ea7fc700 1 -- 172.21.15.52:0/1268637922 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7ff6f400f070 con 0x7ff6f80fbe40 2024-02-19T22:09:05.947 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:05.944+0000 7ff6ea7fc700 1 --2- 172.21.15.52:0/1268637922 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7ff6e002fe10 0x7ff6e00322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:05.947 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:05.944+0000 7ff6ea7fc700 1 -- 172.21.15.52:0/1268637922 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7ff6f4041400 con 0x7ff6f80fbe40 2024-02-19T22:09:05.947 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:05.944+0000 7ff6fd34c700 1 --2- 172.21.15.52:0/1268637922 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7ff6e002fe10 0x7ff6e00322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:05.947 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:05.944+0000 7ff6fd34c700 1 --2- 172.21.15.52:0/1268637922 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7ff6e002fe10 0x7ff6e00322d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7ff6ec0096a0 tx=0x7ff6ec006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:05.947 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:05.945+0000 7ff6ffdb1700 1 -- 172.21.15.52:0/1268637922 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6f8045ad0 con 0x7ff6f80fbe40 2024-02-19T22:09:05.950 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:05.949+0000 7ff6ea7fc700 1 -- 172.21.15.52:0/1268637922 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff6f4016070 con 0x7ff6f80fbe40 2024-02-19T22:09:06.109 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:06.108+0000 7ff6ffdb1700 1 -- 172.21.15.52:0/1268637922 --> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7ff6f8104c60 con 0x7ff6e002fe10 2024-02-19T22:09:06.311 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:06.310+0000 7ff6ea7fc700 1 -- 172.21.15.52:0/1268637922 <== mgr.14114 v2:172.21.15.52:6800/892530721 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff6f8104c60 con 0x7ff6e002fe10 2024-02-19T22:09:06.314 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:06.312+0000 7ff6ffdb1700 1 -- 172.21.15.52:0/1268637922 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7ff6e002fe10 msgr2=0x7ff6e00322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:06.314 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:06.312+0000 7ff6ffdb1700 1 --2- 172.21.15.52:0/1268637922 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7ff6e002fe10 0x7ff6e00322d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7ff6ec0096a0 tx=0x7ff6ec006b40).stop 2024-02-19T22:09:06.314 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:06.312+0000 7ff6ffdb1700 1 -- 172.21.15.52:0/1268637922 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff6f80fbe40 msgr2=0x7ff6f810aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:06.314 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:06.312+0000 7ff6ffdb1700 1 --2- 172.21.15.52:0/1268637922 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff6f80fbe40 0x7ff6f810aba0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7ff6f4002f40 tx=0x7ff6f4006ee0).stop 2024-02-19T22:09:06.314 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:06.312+0000 7ff6ffdb1700 1 -- 172.21.15.52:0/1268637922 shutdown_connections 2024-02-19T22:09:06.314 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:06.312+0000 7ff6ffdb1700 1 --2- 172.21.15.52:0/1268637922 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7ff6e002fe10 0x7ff6e00322d0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:06.314 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:06.312+0000 7ff6ffdb1700 1 --2- 172.21.15.52:0/1268637922 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff6f80fbe40 0x7ff6f810aba0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:06.314 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:06.312+0000 7ff6ffdb1700 1 -- 172.21.15.52:0/1268637922 >> 172.21.15.52:0/1268637922 conn(0x7ff6f80f7640 msgr2=0x7ff6f80f9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:06.314 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:06.312+0000 7ff6ffdb1700 1 -- 172.21.15.52:0/1268637922 shutdown_connections 2024-02-19T22:09:06.314 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:06.312+0000 7ff6ffdb1700 1 -- 172.21.15.52:0/1268637922 wait complete. 2024-02-19T22:09:06.832 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi052 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphvym4zi3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp6ou7w4ul:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2024-02-19T22:09:07.395 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.388+0000 7f6e95ec8700 1 Processor -- start 2024-02-19T22:09:07.395 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.388+0000 7f6e95ec8700 1 -- start start 2024-02-19T22:09:07.395 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.388+0000 7f6e95ec8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6e90072500 0x7f6e90070aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:07.395 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.388+0000 7f6e95ec8700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e90072920 con 0x7f6e90072500 2024-02-19T22:09:07.395 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.389+0000 7f6e8f7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6e90072500 0x7f6e90070aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:07.395 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.389+0000 7f6e8f7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6e90072500 0x7f6e90070aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:55810/0 (socket says 172.21.15.52:55810) 2024-02-19T22:09:07.396 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.389+0000 7f6e8f7fe700 1 -- 172.21.15.52:0/2216929426 learned_addr learned my addr 172.21.15.52:0/2216929426 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:09:07.396 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.389+0000 7f6e8f7fe700 1 -- 172.21.15.52:0/2216929426 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e90070f90 con 0x7f6e90072500 2024-02-19T22:09:07.396 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.389+0000 7f6e8f7fe700 1 --2- 172.21.15.52:0/2216929426 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6e90072500 0x7f6e90070aa0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f6e8000b5f0 tx=0x7f6e80006ee0).ready entity=mon.0 client_cookie=5b07bd5ac623b147 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:07.396 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.389+0000 7f6e8e7fc700 1 -- 172.21.15.52:0/2216929426 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e8000d600 con 0x7f6e90072500 2024-02-19T22:09:07.396 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.389+0000 7f6e8e7fc700 1 -- 172.21.15.52:0/2216929426 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f6e8000dbf0 con 0x7f6e90072500 2024-02-19T22:09:07.396 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.389+0000 7f6e8e7fc700 1 -- 172.21.15.52:0/2216929426 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e80011cb0 con 0x7f6e90072500 2024-02-19T22:09:07.396 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.390+0000 7f6e95ec8700 1 -- 172.21.15.52:0/2216929426 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6e90072500 msgr2=0x7f6e90070aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:07.396 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.390+0000 7f6e95ec8700 1 --2- 172.21.15.52:0/2216929426 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6e90072500 0x7f6e90070aa0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f6e8000b5f0 tx=0x7f6e80006ee0).stop 2024-02-19T22:09:07.396 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.390+0000 7f6e95ec8700 1 -- 172.21.15.52:0/2216929426 shutdown_connections 2024-02-19T22:09:07.396 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.390+0000 7f6e95ec8700 1 --2- 172.21.15.52:0/2216929426 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6e90072500 0x7f6e90070aa0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:07.396 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.390+0000 7f6e95ec8700 1 -- 172.21.15.52:0/2216929426 >> 172.21.15.52:0/2216929426 conn(0x7f6e900f76a0 msgr2=0x7f6e900f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:07.396 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.390+0000 7f6e95ec8700 1 -- 172.21.15.52:0/2216929426 shutdown_connections 2024-02-19T22:09:07.397 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.390+0000 7f6e95ec8700 1 -- 172.21.15.52:0/2216929426 wait complete. 2024-02-19T22:09:07.397 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.390+0000 7f6e95ec8700 1 Processor -- start 2024-02-19T22:09:07.397 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.391+0000 7f6e95ec8700 1 -- start start 2024-02-19T22:09:07.397 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.391+0000 7f6e95ec8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6e90072500 0x7f6e9010ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:07.397 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.391+0000 7f6e95ec8700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9010d300 con 0x7f6e90072500 2024-02-19T22:09:07.397 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.391+0000 7f6e8f7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6e90072500 0x7f6e9010ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:07.397 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.391+0000 7f6e8f7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6e90072500 0x7f6e9010ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:55826/0 (socket says 172.21.15.52:55826) 2024-02-19T22:09:07.397 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.391+0000 7f6e8f7fe700 1 -- 172.21.15.52:0/3517375425 learned_addr learned my addr 172.21.15.52:0/3517375425 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:09:07.397 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.391+0000 7f6e8f7fe700 1 -- 172.21.15.52:0/3517375425 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e8000b200 con 0x7f6e90072500 2024-02-19T22:09:07.397 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.391+0000 7f6e8f7fe700 1 --2- 172.21.15.52:0/3517375425 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6e90072500 0x7f6e9010ce10 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f6e80006800 tx=0x7f6e80006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:07.397 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.391+0000 7f6e87fff700 1 -- 172.21.15.52:0/3517375425 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e8001b570 con 0x7f6e90072500 2024-02-19T22:09:07.397 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.391+0000 7f6e95ec8700 1 -- 172.21.15.52:0/3517375425 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e9010d500 con 0x7f6e90072500 2024-02-19T22:09:07.397 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.392+0000 7f6e87fff700 1 -- 172.21.15.52:0/3517375425 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f6e8001bb60 con 0x7f6e90072500 2024-02-19T22:09:07.397 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.392+0000 7f6e87fff700 1 -- 172.21.15.52:0/3517375425 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e8001a940 con 0x7f6e90072500 2024-02-19T22:09:07.397 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.392+0000 7f6e95ec8700 1 -- 172.21.15.52:0/3517375425 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e9010e110 con 0x7f6e90072500 2024-02-19T22:09:07.398 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.393+0000 7f6e95ec8700 1 -- 172.21.15.52:0/3517375425 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e90059100 con 0x7f6e90072500 2024-02-19T22:09:07.398 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.393+0000 7f6e87fff700 1 -- 172.21.15.52:0/3517375425 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f6e8000f070 con 0x7f6e90072500 2024-02-19T22:09:07.398 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.393+0000 7f6e87fff700 1 --2- 172.21.15.52:0/3517375425 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7f6e7002fe10 0x7f6e700322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:07.398 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.393+0000 7f6e87fff700 1 -- 172.21.15.52:0/3517375425 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f6e80041430 con 0x7f6e90072500 2024-02-19T22:09:07.398 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.393+0000 7f6e8effd700 1 --2- 172.21.15.52:0/3517375425 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7f6e7002fe10 0x7f6e700322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:07.398 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.394+0000 7f6e8effd700 1 --2- 172.21.15.52:0/3517375425 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7f6e7002fe10 0x7f6e700322d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f6e780096a0 tx=0x7f6e78006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:07.398 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.397+0000 7f6e87fff700 1 -- 172.21.15.52:0/3517375425 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6e80019c90 con 0x7f6e90072500 2024-02-19T22:09:07.557 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.556+0000 7f6e95ec8700 1 -- 172.21.15.52:0/3517375425 --> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f6e9010e350 con 0x7f6e7002fe10 2024-02-19T22:09:07.557 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.556+0000 7f6e87fff700 1 -- 172.21.15.52:0/3517375425 <== mgr.14114 v2:172.21.15.52:6800/892530721 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f6e9010e350 con 0x7f6e7002fe10 2024-02-19T22:09:07.558 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCeHpDwSV4bWEN4DlspoLjwZAe49WSN4rYbgGQ2aTR1gtmwj5Vpfk2vL8OXNpwOCCYDZTUwvmxVeUgmBG3NMzOK9lVaiOuKcX6prFa5AHxLEg8FMo/4uuiAodKtCEL75Q6JPZ6GPhs06eW4jc7zF8eWjSdSapepUg04jNzjR0O6ZX2f3MXjLP3JZKnhb/gRKNTVWvGa5dB7OwsRwCL96xCZs6tczrHLhZxlshrsTsvS8WVTNpVqBy8jdsbgHdO33mxVDNkgHBQaU3te2qxtovycnI0l84SvtbnMtDDsBQeOecVD0IiBHdk99yEHfNiQhVoe7A/dG89DQnnbxsEU1VkX9gMybPE1pXFTWrAJzFsnXDg+iT+Q7KecCap6WzucNgFNUq82TYfpS9QPijEJ2qw90KnwogdD9E5dKtW7XxqkPbFJ2/bONmR6sQYpUeWyXkDJohAaZdQlKKys0Gv1Pcab80zIuLKAh2/cqgM2rZTnulwgM0CxHGsiokUA2I3U5+c= ceph-56739a54-cf73-11ee-95bb-87774f69a715 2024-02-19T22:09:07.560 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.558+0000 7f6e95ec8700 1 -- 172.21.15.52:0/3517375425 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7f6e7002fe10 msgr2=0x7f6e700322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:07.560 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.558+0000 7f6e95ec8700 1 --2- 172.21.15.52:0/3517375425 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7f6e7002fe10 0x7f6e700322d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f6e780096a0 tx=0x7f6e78006b40).stop 2024-02-19T22:09:07.560 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.558+0000 7f6e95ec8700 1 -- 172.21.15.52:0/3517375425 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6e90072500 msgr2=0x7f6e9010ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:07.560 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.558+0000 7f6e95ec8700 1 --2- 172.21.15.52:0/3517375425 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6e90072500 0x7f6e9010ce10 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f6e80006800 tx=0x7f6e80006ee0).stop 2024-02-19T22:09:07.560 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.558+0000 7f6e95ec8700 1 -- 172.21.15.52:0/3517375425 shutdown_connections 2024-02-19T22:09:07.560 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.558+0000 7f6e95ec8700 1 --2- 172.21.15.52:0/3517375425 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7f6e7002fe10 0x7f6e700322d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:07.561 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.558+0000 7f6e95ec8700 1 --2- 172.21.15.52:0/3517375425 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6e90072500 0x7f6e9010ce10 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:07.561 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.558+0000 7f6e95ec8700 1 -- 172.21.15.52:0/3517375425 >> 172.21.15.52:0/3517375425 conn(0x7f6e900f76a0 msgr2=0x7f6e900f90f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:07.561 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.558+0000 7f6e95ec8700 1 -- 172.21.15.52:0/3517375425 shutdown_connections 2024-02-19T22:09:07.561 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:07.558+0000 7f6e95ec8700 1 -- 172.21.15.52:0/3517375425 wait complete. 2024-02-19T22:09:07.579 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:07 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cephadm 2024-02-19T22:09:06.110495+0000 mgr.smithi052.ghqrtk (mgr.14114) 6 : cephadm [INF] Generating ssh key... 2024-02-19T22:09:07.579 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:07 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:06.309172+0000 mon.smithi052 (mon.0) 52 : audit [INF] from='mgr.14114 172.21.15.52:0/3613687413' entity='mgr.smithi052.ghqrtk' 2024-02-19T22:09:07.579 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:07 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:06.310818+0000 mon.smithi052 (mon.0) 53 : audit [INF] from='mgr.14114 172.21.15.52:0/3613687413' entity='mgr.smithi052.ghqrtk' 2024-02-19T22:09:08.164 INFO:teuthology.orchestra.run.smithi052.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-19T22:09:08.165 INFO:teuthology.orchestra.run.smithi052.stderr:Adding key to root@localhost authorized_keys... 2024-02-19T22:09:08.166 INFO:teuthology.orchestra.run.smithi052.stderr:Adding host smithi052... 2024-02-19T22:09:08.166 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi052 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphvym4zi3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp6ou7w4ul:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi052 172.21.15.52 2024-02-19T22:09:08.447 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:08 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:07.557789+0000 mgr.smithi052.ghqrtk (mgr.14114) 7 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:09:08.701 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:08.699+0000 7f429af41700 1 Processor -- start 2024-02-19T22:09:08.702 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:08.699+0000 7f429af41700 1 -- start start 2024-02-19T22:09:08.702 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:08.700+0000 7f429af41700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4294070430 0x7f42940fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:08.702 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:08.700+0000 7f429af41700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42940fef40 con 0x7f4294070430 2024-02-19T22:09:08.702 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:08.700+0000 7f4298cdd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4294070430 0x7f42940fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:08.702 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:08.700+0000 7f4298cdd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4294070430 0x7f42940fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:55836/0 (socket says 172.21.15.52:55836) 2024-02-19T22:09:08.702 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:08.700+0000 7f4298cdd700 1 -- 172.21.15.52:0/2383617673 learned_addr learned my addr 172.21.15.52:0/2383617673 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:09:08.706 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:08.700+0000 7f4298cdd700 1 -- 172.21.15.52:0/2383617673 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42940ff080 con 0x7f4294070430 2024-02-19T22:09:08.706 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:08.700+0000 7f4298cdd700 1 --2- 172.21.15.52:0/2383617673 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4294070430 0x7f42940fea50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f428800b5f0 tx=0x7f4288006ee0).ready entity=mon.0 client_cookie=c60a3139762602c0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:08.706 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:08.700+0000 7f42937fe700 1 -- 172.21.15.52:0/2383617673 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f428800d600 con 0x7f4294070430 2024-02-19T22:09:08.706 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:08.701+0000 7f42937fe700 1 -- 172.21.15.52:0/2383617673 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f428800dbf0 con 0x7f4294070430 2024-02-19T22:09:08.706 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:08.701+0000 7f42937fe700 1 -- 172.21.15.52:0/2383617673 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4288011cb0 con 0x7f4294070430 2024-02-19T22:09:08.706 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:08.701+0000 7f429af41700 1 -- 172.21.15.52:0/2383617673 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4294070430 msgr2=0x7f42940fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:08.706 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:08.701+0000 7f429af41700 1 --2- 172.21.15.52:0/2383617673 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4294070430 0x7f42940fea50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f428800b5f0 tx=0x7f4288006ee0).stop 2024-02-19T22:09:08.706 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:08.701+0000 7f429af41700 1 -- 172.21.15.52:0/2383617673 shutdown_connections 2024-02-19T22:09:08.706 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:08.701+0000 7f429af41700 1 --2- 172.21.15.52:0/2383617673 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4294070430 0x7f42940fea50 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:08.706 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:08.701+0000 7f429af41700 1 -- 172.21.15.52:0/2383617673 >> 172.21.15.52:0/2383617673 conn(0x7f42940f6530 msgr2=0x7f42940f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:08.706 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:08.701+0000 7f429af41700 1 -- 172.21.15.52:0/2383617673 shutdown_connections 2024-02-19T22:09:08.707 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:08.701+0000 7f429af41700 1 -- 172.21.15.52:0/2383617673 wait complete. 2024-02-19T22:09:08.707 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:08.702+0000 7f429af41700 1 Processor -- start 2024-02-19T22:09:08.707 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:08.702+0000 7f429af41700 1 -- start start 2024-02-19T22:09:08.707 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:08.702+0000 7f429af41700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4294070430 0x7f429410ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:08.707 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:08.702+0000 7f429af41700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f429410b020 con 0x7f4294070430 2024-02-19T22:09:08.707 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:08.702+0000 7f4298cdd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4294070430 0x7f429410ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:08.707 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:08.702+0000 7f4298cdd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4294070430 0x7f429410ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:55844/0 (socket says 172.21.15.52:55844) 2024-02-19T22:09:08.707 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:08.702+0000 7f4298cdd700 1 -- 172.21.15.52:0/3762601329 learned_addr learned my addr 172.21.15.52:0/3762601329 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:09:08.707 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:08.703+0000 7f4298cdd700 1 -- 172.21.15.52:0/3762601329 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f428800b200 con 0x7f4294070430 2024-02-19T22:09:08.707 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:08.703+0000 7f4298cdd700 1 --2- 172.21.15.52:0/3762601329 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4294070430 0x7f429410ab30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f4288006800 tx=0x7f4288006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:08.707 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:08.703+0000 7f42917fa700 1 -- 172.21.15.52:0/3762601329 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f428801b570 con 0x7f4294070430 2024-02-19T22:09:08.707 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:08.703+0000 7f42917fa700 1 -- 172.21.15.52:0/3762601329 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f428801bb60 con 0x7f4294070430 2024-02-19T22:09:08.707 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:08.703+0000 7f429af41700 1 -- 172.21.15.52:0/3762601329 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f429410b220 con 0x7f4294070430 2024-02-19T22:09:08.707 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:08.703+0000 7f42917fa700 1 -- 172.21.15.52:0/3762601329 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f428801a9f0 con 0x7f4294070430 2024-02-19T22:09:08.707 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:08.703+0000 7f429af41700 1 -- 172.21.15.52:0/3762601329 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f429410b6b0 con 0x7f4294070430 2024-02-19T22:09:08.708 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:08.704+0000 7f42917fa700 1 -- 172.21.15.52:0/3762601329 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f428800f070 con 0x7f4294070430 2024-02-19T22:09:08.708 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:08.704+0000 7f429af41700 1 -- 172.21.15.52:0/3762601329 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4294045ad0 con 0x7f4294070430 2024-02-19T22:09:08.708 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:08.704+0000 7f42917fa700 1 --2- 172.21.15.52:0/3762601329 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7f427802fe70 0x7f4278032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:08.708 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:08.704+0000 7f42917fa700 1 -- 172.21.15.52:0/3762601329 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f4288041410 con 0x7f4294070430 2024-02-19T22:09:08.708 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:08.704+0000 7f4293fff700 1 --2- 172.21.15.52:0/3762601329 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7f427802fe70 0x7f4278032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:08.708 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:08.705+0000 7f4293fff700 1 --2- 172.21.15.52:0/3762601329 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7f427802fe70 0x7f4278032330 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f42800096a0 tx=0x7f4280006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:08.709 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:08.708+0000 7f42917fa700 1 -- 172.21.15.52:0/3762601329 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4288022d70 con 0x7f4294070430 2024-02-19T22:09:08.894 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:08.893+0000 7f429af41700 1 -- 172.21.15.52:0/3762601329 --> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi052", "addr": "172.21.15.52", "target": ["mon-mgr", ""]}) v1 -- 0x7f429410bf10 con 0x7f427802fe70 2024-02-19T22:09:09.514 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: Added host 'smithi052' 2024-02-19T22:09:09.514 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:09.512+0000 7f42917fa700 1 -- 172.21.15.52:0/3762601329 <== mgr.14114 v2:172.21.15.52:6800/892530721 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f429410bf10 con 0x7f427802fe70 2024-02-19T22:09:09.516 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:09.514+0000 7f429af41700 1 -- 172.21.15.52:0/3762601329 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7f427802fe70 msgr2=0x7f4278032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:09.516 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:09.514+0000 7f429af41700 1 --2- 172.21.15.52:0/3762601329 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7f427802fe70 0x7f4278032330 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f42800096a0 tx=0x7f4280006b40).stop 2024-02-19T22:09:09.516 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:09.514+0000 7f429af41700 1 -- 172.21.15.52:0/3762601329 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4294070430 msgr2=0x7f429410ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:09.516 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:09.514+0000 7f429af41700 1 --2- 172.21.15.52:0/3762601329 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4294070430 0x7f429410ab30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f4288006800 tx=0x7f4288006ee0).stop 2024-02-19T22:09:09.516 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:09.514+0000 7f429af41700 1 -- 172.21.15.52:0/3762601329 shutdown_connections 2024-02-19T22:09:09.516 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:09.514+0000 7f429af41700 1 --2- 172.21.15.52:0/3762601329 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7f427802fe70 0x7f4278032330 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:09.516 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:09.514+0000 7f429af41700 1 --2- 172.21.15.52:0/3762601329 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4294070430 0x7f429410ab30 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:09.516 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:09.514+0000 7f429af41700 1 -- 172.21.15.52:0/3762601329 >> 172.21.15.52:0/3762601329 conn(0x7f42940f6530 msgr2=0x7f42940f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:09.516 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:09.514+0000 7f429af41700 1 -- 172.21.15.52:0/3762601329 shutdown_connections 2024-02-19T22:09:09.516 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:09.514+0000 7f429af41700 1 -- 172.21.15.52:0/3762601329 wait complete. 2024-02-19T22:09:10.213 INFO:teuthology.orchestra.run.smithi052.stderr:Deploying mon service with default placement... 2024-02-19T22:09:10.213 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi052 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphvym4zi3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp6ou7w4ul:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2024-02-19T22:09:10.780 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.777+0000 7f62e8c20700 1 Processor -- start 2024-02-19T22:09:10.780 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.777+0000 7f62e8c20700 1 -- start start 2024-02-19T22:09:10.780 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.777+0000 7f62e8c20700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62e4070430 0x7f62e40fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:10.780 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.777+0000 7f62e8c20700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e40fef40 con 0x7f62e4070430 2024-02-19T22:09:10.780 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.778+0000 7f62e259c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62e4070430 0x7f62e40fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:10.780 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.778+0000 7f62e259c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62e4070430 0x7f62e40fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:55846/0 (socket says 172.21.15.52:55846) 2024-02-19T22:09:10.780 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.778+0000 7f62e259c700 1 -- 172.21.15.52:0/1864541464 learned_addr learned my addr 172.21.15.52:0/1864541464 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:09:10.780 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.778+0000 7f62e259c700 1 -- 172.21.15.52:0/1864541464 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62e40ff080 con 0x7f62e4070430 2024-02-19T22:09:10.781 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.778+0000 7f62e259c700 1 --2- 172.21.15.52:0/1864541464 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62e4070430 0x7f62e40fea50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f62d800b5f0 tx=0x7f62d8006ee0).ready entity=mon.0 client_cookie=60898a1cc9702d01 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:10.781 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.778+0000 7f62e159a700 1 -- 172.21.15.52:0/1864541464 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62d800d600 con 0x7f62e4070430 2024-02-19T22:09:10.781 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.778+0000 7f62e159a700 1 -- 172.21.15.52:0/1864541464 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f62d800dbf0 con 0x7f62e4070430 2024-02-19T22:09:10.781 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.778+0000 7f62e159a700 1 -- 172.21.15.52:0/1864541464 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62d8011cb0 con 0x7f62e4070430 2024-02-19T22:09:10.781 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.779+0000 7f62e8c20700 1 -- 172.21.15.52:0/1864541464 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62e4070430 msgr2=0x7f62e40fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:10.781 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.779+0000 7f62e8c20700 1 --2- 172.21.15.52:0/1864541464 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62e4070430 0x7f62e40fea50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f62d800b5f0 tx=0x7f62d8006ee0).stop 2024-02-19T22:09:10.783 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.779+0000 7f62e8c20700 1 -- 172.21.15.52:0/1864541464 shutdown_connections 2024-02-19T22:09:10.783 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.779+0000 7f62e8c20700 1 --2- 172.21.15.52:0/1864541464 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62e4070430 0x7f62e40fea50 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:10.784 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.779+0000 7f62e8c20700 1 -- 172.21.15.52:0/1864541464 >> 172.21.15.52:0/1864541464 conn(0x7f62e40f6530 msgr2=0x7f62e40f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:10.784 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.779+0000 7f62e8c20700 1 -- 172.21.15.52:0/1864541464 shutdown_connections 2024-02-19T22:09:10.784 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.779+0000 7f62e8c20700 1 -- 172.21.15.52:0/1864541464 wait complete. 2024-02-19T22:09:10.784 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.780+0000 7f62e8c20700 1 Processor -- start 2024-02-19T22:09:10.784 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.780+0000 7f62e8c20700 1 -- start start 2024-02-19T22:09:10.784 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.780+0000 7f62e8c20700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62e4070430 0x7f62e40fc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:10.784 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.780+0000 7f62e8c20700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e40fa950 con 0x7f62e4070430 2024-02-19T22:09:10.784 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.780+0000 7f62e259c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62e4070430 0x7f62e40fc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:10.784 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.780+0000 7f62e259c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62e4070430 0x7f62e40fc4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:55862/0 (socket says 172.21.15.52:55862) 2024-02-19T22:09:10.784 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.780+0000 7f62e259c700 1 -- 172.21.15.52:0/1924389366 learned_addr learned my addr 172.21.15.52:0/1924389366 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:09:10.784 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.781+0000 7f62e259c700 1 -- 172.21.15.52:0/1924389366 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d800b200 con 0x7f62e4070430 2024-02-19T22:09:10.784 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.781+0000 7f62e259c700 1 --2- 172.21.15.52:0/1924389366 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62e4070430 0x7f62e40fc4d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f62d8006800 tx=0x7f62d8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:10.784 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.781+0000 7f62d2ffd700 1 -- 172.21.15.52:0/1924389366 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62d801b570 con 0x7f62e4070430 2024-02-19T22:09:10.784 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.781+0000 7f62d2ffd700 1 -- 172.21.15.52:0/1924389366 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f62d801bb60 con 0x7f62e4070430 2024-02-19T22:09:10.785 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.781+0000 7f62e8c20700 1 -- 172.21.15.52:0/1924389366 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62e40faaf0 con 0x7f62e4070430 2024-02-19T22:09:10.785 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.781+0000 7f62d2ffd700 1 -- 172.21.15.52:0/1924389366 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62d801a9f0 con 0x7f62e4070430 2024-02-19T22:09:10.785 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.781+0000 7f62e8c20700 1 -- 172.21.15.52:0/1924389366 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62e40faf80 con 0x7f62e4070430 2024-02-19T22:09:10.785 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.782+0000 7f62d2ffd700 1 -- 172.21.15.52:0/1924389366 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f62d800f070 con 0x7f62e4070430 2024-02-19T22:09:10.785 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.782+0000 7f62d2ffd700 1 --2- 172.21.15.52:0/1924389366 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7f62c802fe70 0x7f62c8032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:10.785 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.782+0000 7f62d2ffd700 1 -- 172.21.15.52:0/1924389366 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f62d8041420 con 0x7f62e4070430 2024-02-19T22:09:10.785 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.782+0000 7f62e8c20700 1 -- 172.21.15.52:0/1924389366 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62e4045ad0 con 0x7f62e4070430 2024-02-19T22:09:10.785 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.782+0000 7f62e1d9b700 1 --2- 172.21.15.52:0/1924389366 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7f62c802fe70 0x7f62c8032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:10.785 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.782+0000 7f62e1d9b700 1 --2- 172.21.15.52:0/1924389366 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7f62c802fe70 0x7f62c8032330 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f62d40096a0 tx=0x7f62d4006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:10.787 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.786+0000 7f62d2ffd700 1 -- 172.21.15.52:0/1924389366 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f62d8016070 con 0x7f62e4070430 2024-02-19T22:09:10.841 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:10 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:08.894838+0000 mgr.smithi052.ghqrtk (mgr.14114) 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi052", "addr": "172.21.15.52", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:09:10.841 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:10 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:09.512775+0000 mon.smithi052 (mon.0) 54 : audit [INF] from='mgr.14114 172.21.15.52:0/3613687413' entity='mgr.smithi052.ghqrtk' 2024-02-19T22:09:10.841 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:10 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cephadm 2024-02-19T22:09:09.513099+0000 mgr.smithi052.ghqrtk (mgr.14114) 9 : cephadm [INF] Added host smithi052 2024-02-19T22:09:10.842 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:10 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:09.513352+0000 mon.smithi052 (mon.0) 55 : audit [DBG] from='mgr.14114 172.21.15.52:0/3613687413' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T22:09:10.971 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.970+0000 7f62e8c20700 1 -- 172.21.15.52:0/1924389366 --> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f62e4104b60 con 0x7f62c802fe70 2024-02-19T22:09:10.975 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.974+0000 7f62d2ffd700 1 -- 172.21.15.52:0/1924389366 <== mgr.14114 v2:172.21.15.52:6800/892530721 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f62e4104b60 con 0x7f62c802fe70 2024-02-19T22:09:10.975 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: Scheduled mon update... 2024-02-19T22:09:10.977 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.976+0000 7f62e8c20700 1 -- 172.21.15.52:0/1924389366 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7f62c802fe70 msgr2=0x7f62c8032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:10.977 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.976+0000 7f62e8c20700 1 --2- 172.21.15.52:0/1924389366 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7f62c802fe70 0x7f62c8032330 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f62d40096a0 tx=0x7f62d4006b40).stop 2024-02-19T22:09:10.978 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.976+0000 7f62e8c20700 1 -- 172.21.15.52:0/1924389366 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62e4070430 msgr2=0x7f62e40fc4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:10.978 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.976+0000 7f62e8c20700 1 --2- 172.21.15.52:0/1924389366 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62e4070430 0x7f62e40fc4d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f62d8006800 tx=0x7f62d8006ee0).stop 2024-02-19T22:09:10.978 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.976+0000 7f62e8c20700 1 -- 172.21.15.52:0/1924389366 shutdown_connections 2024-02-19T22:09:10.978 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.976+0000 7f62e8c20700 1 --2- 172.21.15.52:0/1924389366 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7f62c802fe70 0x7f62c8032330 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:10.978 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.976+0000 7f62e8c20700 1 --2- 172.21.15.52:0/1924389366 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62e4070430 0x7f62e40fc4d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:10.978 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.976+0000 7f62e8c20700 1 -- 172.21.15.52:0/1924389366 >> 172.21.15.52:0/1924389366 conn(0x7f62e40f6530 msgr2=0x7f62e40f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:10.978 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.977+0000 7f62e8c20700 1 -- 172.21.15.52:0/1924389366 shutdown_connections 2024-02-19T22:09:10.978 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:10.977+0000 7f62e8c20700 1 -- 172.21.15.52:0/1924389366 wait complete. 2024-02-19T22:09:11.570 INFO:teuthology.orchestra.run.smithi052.stderr:Deploying mgr service with default placement... 2024-02-19T22:09:11.570 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi052 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphvym4zi3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp6ou7w4ul:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2024-02-19T22:09:12.092 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:11 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:10.972419+0000 mgr.smithi052.ghqrtk (mgr.14114) 10 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:09:12.092 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:11 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cephadm 2024-02-19T22:09:10.973164+0000 mgr.smithi052.ghqrtk (mgr.14114) 11 : cephadm [INF] Saving service mon spec with placement count:5 2024-02-19T22:09:12.092 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:11 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:10.975359+0000 mon.smithi052 (mon.0) 56 : audit [INF] from='mgr.14114 172.21.15.52:0/3613687413' entity='mgr.smithi052.ghqrtk' 2024-02-19T22:09:12.152 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.149+0000 7f0f3bfa7700 1 Processor -- start 2024-02-19T22:09:12.152 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.149+0000 7f0f3bfa7700 1 -- start start 2024-02-19T22:09:12.152 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.150+0000 7f0f3bfa7700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f340fad90 0x7f0f340fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:12.152 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.150+0000 7f0f3bfa7700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f340fb6a0 con 0x7f0f340fad90 2024-02-19T22:09:12.152 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.150+0000 7f0f39d43700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f340fad90 0x7f0f340fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:12.152 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.150+0000 7f0f39d43700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f340fad90 0x7f0f340fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:55872/0 (socket says 172.21.15.52:55872) 2024-02-19T22:09:12.152 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.150+0000 7f0f39d43700 1 -- 172.21.15.52:0/3893313315 learned_addr learned my addr 172.21.15.52:0/3893313315 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:09:12.152 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.150+0000 7f0f39d43700 1 -- 172.21.15.52:0/3893313315 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f340fb7e0 con 0x7f0f340fad90 2024-02-19T22:09:12.152 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.150+0000 7f0f39d43700 1 --2- 172.21.15.52:0/3893313315 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f340fad90 0x7f0f340fb1b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f0f3000b5f0 tx=0x7f0f30006ee0).ready entity=mon.0 client_cookie=4116090196fba84b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:12.152 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.150+0000 7f0f38d41700 1 -- 172.21.15.52:0/3893313315 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f3000d600 con 0x7f0f340fad90 2024-02-19T22:09:12.152 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.150+0000 7f0f38d41700 1 -- 172.21.15.52:0/3893313315 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f0f3000dbf0 con 0x7f0f340fad90 2024-02-19T22:09:12.152 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.150+0000 7f0f38d41700 1 -- 172.21.15.52:0/3893313315 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f30011cb0 con 0x7f0f340fad90 2024-02-19T22:09:12.155 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.151+0000 7f0f3bfa7700 1 -- 172.21.15.52:0/3893313315 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f340fad90 msgr2=0x7f0f340fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:12.155 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.151+0000 7f0f3bfa7700 1 --2- 172.21.15.52:0/3893313315 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f340fad90 0x7f0f340fb1b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f0f3000b5f0 tx=0x7f0f30006ee0).stop 2024-02-19T22:09:12.156 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.151+0000 7f0f3bfa7700 1 -- 172.21.15.52:0/3893313315 shutdown_connections 2024-02-19T22:09:12.156 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.151+0000 7f0f3bfa7700 1 --2- 172.21.15.52:0/3893313315 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f340fad90 0x7f0f340fb1b0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:12.156 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.151+0000 7f0f3bfa7700 1 -- 172.21.15.52:0/3893313315 >> 172.21.15.52:0/3893313315 conn(0x7f0f340f6530 msgr2=0x7f0f340f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:12.156 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.151+0000 7f0f3bfa7700 1 -- 172.21.15.52:0/3893313315 shutdown_connections 2024-02-19T22:09:12.156 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.151+0000 7f0f3bfa7700 1 -- 172.21.15.52:0/3893313315 wait complete. 2024-02-19T22:09:12.156 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.152+0000 7f0f3bfa7700 1 Processor -- start 2024-02-19T22:09:12.156 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.152+0000 7f0f3bfa7700 1 -- start start 2024-02-19T22:09:12.156 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.152+0000 7f0f3bfa7700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f340fad90 0x7f0f3410d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:12.156 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.152+0000 7f0f3bfa7700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f34070450 con 0x7f0f340fad90 2024-02-19T22:09:12.156 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.152+0000 7f0f39d43700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f340fad90 0x7f0f3410d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:12.156 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.152+0000 7f0f39d43700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f340fad90 0x7f0f3410d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:55888/0 (socket says 172.21.15.52:55888) 2024-02-19T22:09:12.156 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.152+0000 7f0f39d43700 1 -- 172.21.15.52:0/1497016735 learned_addr learned my addr 172.21.15.52:0/1497016735 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:09:12.157 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.153+0000 7f0f39d43700 1 -- 172.21.15.52:0/1497016735 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f3000b200 con 0x7f0f340fad90 2024-02-19T22:09:12.157 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.153+0000 7f0f39d43700 1 --2- 172.21.15.52:0/1497016735 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f340fad90 0x7f0f3410d860 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f0f30011ea0 tx=0x7f0f30006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:12.157 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.153+0000 7f0f267fc700 1 -- 172.21.15.52:0/1497016735 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f3001b570 con 0x7f0f340fad90 2024-02-19T22:09:12.157 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.153+0000 7f0f267fc700 1 -- 172.21.15.52:0/1497016735 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f0f3001bb60 con 0x7f0f340fad90 2024-02-19T22:09:12.157 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.153+0000 7f0f3bfa7700 1 -- 172.21.15.52:0/1497016735 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f34070650 con 0x7f0f340fad90 2024-02-19T22:09:12.157 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.153+0000 7f0f267fc700 1 -- 172.21.15.52:0/1497016735 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f3001a8d0 con 0x7f0f340fad90 2024-02-19T22:09:12.157 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.153+0000 7f0f3bfa7700 1 -- 172.21.15.52:0/1497016735 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f3410df10 con 0x7f0f340fad90 2024-02-19T22:09:12.157 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.154+0000 7f0f267fc700 1 -- 172.21.15.52:0/1497016735 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f0f3000f070 con 0x7f0f340fad90 2024-02-19T22:09:12.157 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.154+0000 7f0f3bfa7700 1 -- 172.21.15.52:0/1497016735 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f34045ad0 con 0x7f0f340fad90 2024-02-19T22:09:12.157 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.154+0000 7f0f267fc700 1 --2- 172.21.15.52:0/1497016735 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7f0f1c02fe10 0x7f0f1c0322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:12.157 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.154+0000 7f0f267fc700 1 -- 172.21.15.52:0/1497016735 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f0f300414b0 con 0x7f0f340fad90 2024-02-19T22:09:12.157 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.154+0000 7f0f39542700 1 --2- 172.21.15.52:0/1497016735 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7f0f1c02fe10 0x7f0f1c0322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:12.157 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.155+0000 7f0f39542700 1 --2- 172.21.15.52:0/1497016735 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7f0f1c02fe10 0x7f0f1c0322d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f0f280096a0 tx=0x7f0f28006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:12.159 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.158+0000 7f0f267fc700 1 -- 172.21.15.52:0/1497016735 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0f300224f0 con 0x7f0f340fad90 2024-02-19T22:09:12.345 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.344+0000 7f0f3bfa7700 1 -- 172.21.15.52:0/1497016735 --> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f0f34057f10 con 0x7f0f1c02fe10 2024-02-19T22:09:12.348 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.347+0000 7f0f267fc700 1 -- 172.21.15.52:0/1497016735 <== mgr.14114 v2:172.21.15.52:6800/892530721 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f0f34057f10 con 0x7f0f1c02fe10 2024-02-19T22:09:12.348 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-02-19T22:09:12.351 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.349+0000 7f0f3bfa7700 1 -- 172.21.15.52:0/1497016735 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7f0f1c02fe10 msgr2=0x7f0f1c0322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:12.351 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.349+0000 7f0f3bfa7700 1 --2- 172.21.15.52:0/1497016735 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7f0f1c02fe10 0x7f0f1c0322d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f0f280096a0 tx=0x7f0f28006b40).stop 2024-02-19T22:09:12.351 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.349+0000 7f0f3bfa7700 1 -- 172.21.15.52:0/1497016735 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f340fad90 msgr2=0x7f0f3410d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:12.351 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.349+0000 7f0f3bfa7700 1 --2- 172.21.15.52:0/1497016735 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f340fad90 0x7f0f3410d860 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f0f30011ea0 tx=0x7f0f30006ee0).stop 2024-02-19T22:09:12.351 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.349+0000 7f0f3bfa7700 1 -- 172.21.15.52:0/1497016735 shutdown_connections 2024-02-19T22:09:12.351 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.349+0000 7f0f3bfa7700 1 --2- 172.21.15.52:0/1497016735 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7f0f1c02fe10 0x7f0f1c0322d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:12.351 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.349+0000 7f0f3bfa7700 1 --2- 172.21.15.52:0/1497016735 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f340fad90 0x7f0f3410d860 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:12.351 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.349+0000 7f0f3bfa7700 1 -- 172.21.15.52:0/1497016735 >> 172.21.15.52:0/1497016735 conn(0x7f0f340f6530 msgr2=0x7f0f340f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:12.351 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.349+0000 7f0f3bfa7700 1 -- 172.21.15.52:0/1497016735 shutdown_connections 2024-02-19T22:09:12.352 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:12.349+0000 7f0f3bfa7700 1 -- 172.21.15.52:0/1497016735 wait complete. 2024-02-19T22:09:12.910 INFO:teuthology.orchestra.run.smithi052.stderr:Deploying crash service with default placement... 2024-02-19T22:09:12.910 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi052 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphvym4zi3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp6ou7w4ul:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2024-02-19T22:09:13.499 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.494+0000 7fa69bc3a700 1 Processor -- start 2024-02-19T22:09:13.499 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.494+0000 7fa69bc3a700 1 -- start start 2024-02-19T22:09:13.499 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.494+0000 7fa69bc3a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa694070390 0x7fa694100ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:13.499 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.494+0000 7fa69bc3a700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6940fc8e0 con 0x7fa694070390 2024-02-19T22:09:13.499 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.495+0000 7fa6999d6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa694070390 0x7fa694100ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:13.499 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.495+0000 7fa6999d6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa694070390 0x7fa694100ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56518/0 (socket says 172.21.15.52:56518) 2024-02-19T22:09:13.499 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.495+0000 7fa6999d6700 1 -- 172.21.15.52:0/332387050 learned_addr learned my addr 172.21.15.52:0/332387050 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:09:13.499 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.495+0000 7fa6999d6700 1 -- 172.21.15.52:0/332387050 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6941013e0 con 0x7fa694070390 2024-02-19T22:09:13.500 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.495+0000 7fa6999d6700 1 --2- 172.21.15.52:0/332387050 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa694070390 0x7fa694100ef0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fa68400b5f0 tx=0x7fa684006ee0).ready entity=mon.0 client_cookie=24c43b250afdb0b5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:13.500 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.495+0000 7fa6989d4700 1 -- 172.21.15.52:0/332387050 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa68400d600 con 0x7fa694070390 2024-02-19T22:09:13.500 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.495+0000 7fa6989d4700 1 -- 172.21.15.52:0/332387050 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fa68400dbf0 con 0x7fa694070390 2024-02-19T22:09:13.500 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.495+0000 7fa6989d4700 1 -- 172.21.15.52:0/332387050 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa684011cb0 con 0x7fa694070390 2024-02-19T22:09:13.500 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.496+0000 7fa69bc3a700 1 -- 172.21.15.52:0/332387050 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa694070390 msgr2=0x7fa694100ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:13.500 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.496+0000 7fa69bc3a700 1 --2- 172.21.15.52:0/332387050 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa694070390 0x7fa694100ef0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fa68400b5f0 tx=0x7fa684006ee0).stop 2024-02-19T22:09:13.500 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.496+0000 7fa69bc3a700 1 -- 172.21.15.52:0/332387050 shutdown_connections 2024-02-19T22:09:13.500 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.496+0000 7fa69bc3a700 1 --2- 172.21.15.52:0/332387050 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa694070390 0x7fa694100ef0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:13.500 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.496+0000 7fa69bc3a700 1 -- 172.21.15.52:0/332387050 >> 172.21.15.52:0/332387050 conn(0x7fa6940f6530 msgr2=0x7fa6940f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:13.500 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.496+0000 7fa69bc3a700 1 -- 172.21.15.52:0/332387050 shutdown_connections 2024-02-19T22:09:13.500 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.496+0000 7fa69bc3a700 1 -- 172.21.15.52:0/332387050 wait complete. 2024-02-19T22:09:13.500 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.497+0000 7fa69bc3a700 1 Processor -- start 2024-02-19T22:09:13.500 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.497+0000 7fa69bc3a700 1 -- start start 2024-02-19T22:09:13.500 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.497+0000 7fa69bc3a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa694070390 0x7fa69410cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:13.501 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.497+0000 7fa69bc3a700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa69410d290 con 0x7fa694070390 2024-02-19T22:09:13.501 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.497+0000 7fa6999d6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa694070390 0x7fa69410cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:13.501 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.497+0000 7fa6999d6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa694070390 0x7fa69410cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56532/0 (socket says 172.21.15.52:56532) 2024-02-19T22:09:13.501 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.497+0000 7fa6999d6700 1 -- 172.21.15.52:0/923916420 learned_addr learned my addr 172.21.15.52:0/923916420 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:09:13.501 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.497+0000 7fa6999d6700 1 -- 172.21.15.52:0/923916420 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa68400b200 con 0x7fa694070390 2024-02-19T22:09:13.501 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.497+0000 7fa6999d6700 1 --2- 172.21.15.52:0/923916420 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa694070390 0x7fa69410cda0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fa68400df90 tx=0x7fa684006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:13.501 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.497+0000 7fa6827fc700 1 -- 172.21.15.52:0/923916420 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa68401b410 con 0x7fa694070390 2024-02-19T22:09:13.501 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.498+0000 7fa6827fc700 1 -- 172.21.15.52:0/923916420 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fa68401ba00 con 0x7fa694070390 2024-02-19T22:09:13.501 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.498+0000 7fa69bc3a700 1 -- 172.21.15.52:0/923916420 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa69410d490 con 0x7fa694070390 2024-02-19T22:09:13.501 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.498+0000 7fa6827fc700 1 -- 172.21.15.52:0/923916420 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa68401aa30 con 0x7fa694070390 2024-02-19T22:09:13.502 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.498+0000 7fa69bc3a700 1 -- 172.21.15.52:0/923916420 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa69410e0d0 con 0x7fa694070390 2024-02-19T22:09:13.502 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.498+0000 7fa6827fc700 1 -- 172.21.15.52:0/923916420 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fa68400f070 con 0x7fa694070390 2024-02-19T22:09:13.502 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.499+0000 7fa6827fc700 1 --2- 172.21.15.52:0/923916420 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7fa67c02fe10 0x7fa67c0322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:13.502 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.499+0000 7fa6827fc700 1 -- 172.21.15.52:0/923916420 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fa684041170 con 0x7fa694070390 2024-02-19T22:09:13.502 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.499+0000 7fa69bc3a700 1 -- 172.21.15.52:0/923916420 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa69410d920 con 0x7fa694070390 2024-02-19T22:09:13.503 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.499+0000 7fa6991d5700 1 --2- 172.21.15.52:0/923916420 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7fa67c02fe10 0x7fa67c0322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:13.503 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.499+0000 7fa6991d5700 1 --2- 172.21.15.52:0/923916420 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7fa67c02fe10 0x7fa67c0322d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fa6900096a0 tx=0x7fa690006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:13.504 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.503+0000 7fa6827fc700 1 -- 172.21.15.52:0/923916420 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa684016070 con 0x7fa694070390 2024-02-19T22:09:13.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:13 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:12.346080+0000 mgr.smithi052.ghqrtk (mgr.14114) 12 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:09:13.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:13 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cephadm 2024-02-19T22:09:12.346613+0000 mgr.smithi052.ghqrtk (mgr.14114) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2024-02-19T22:09:13.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:13 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:12.348646+0000 mon.smithi052 (mon.0) 57 : audit [INF] from='mgr.14114 172.21.15.52:0/3613687413' entity='mgr.smithi052.ghqrtk' 2024-02-19T22:09:13.693 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.692+0000 7fa69bc3a700 1 -- 172.21.15.52:0/923916420 --> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7fa694102180 con 0x7fa67c02fe10 2024-02-19T22:09:13.696 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.695+0000 7fa6827fc700 1 -- 172.21.15.52:0/923916420 <== mgr.14114 v2:172.21.15.52:6800/892530721 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7fa694102180 con 0x7fa67c02fe10 2024-02-19T22:09:13.696 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: Scheduled crash update... 2024-02-19T22:09:13.699 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.697+0000 7fa69bc3a700 1 -- 172.21.15.52:0/923916420 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7fa67c02fe10 msgr2=0x7fa67c0322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:13.699 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.697+0000 7fa69bc3a700 1 --2- 172.21.15.52:0/923916420 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7fa67c02fe10 0x7fa67c0322d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fa6900096a0 tx=0x7fa690006b40).stop 2024-02-19T22:09:13.699 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.697+0000 7fa69bc3a700 1 -- 172.21.15.52:0/923916420 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa694070390 msgr2=0x7fa69410cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:13.699 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.697+0000 7fa69bc3a700 1 --2- 172.21.15.52:0/923916420 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa694070390 0x7fa69410cda0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fa68400df90 tx=0x7fa684006ee0).stop 2024-02-19T22:09:13.699 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.697+0000 7fa69bc3a700 1 -- 172.21.15.52:0/923916420 shutdown_connections 2024-02-19T22:09:13.699 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.697+0000 7fa69bc3a700 1 --2- 172.21.15.52:0/923916420 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7fa67c02fe10 0x7fa67c0322d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:13.699 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.697+0000 7fa69bc3a700 1 --2- 172.21.15.52:0/923916420 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa694070390 0x7fa69410cda0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:13.699 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.697+0000 7fa69bc3a700 1 -- 172.21.15.52:0/923916420 >> 172.21.15.52:0/923916420 conn(0x7fa6940f6530 msgr2=0x7fa6940f7f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:13.699 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.698+0000 7fa69bc3a700 1 -- 172.21.15.52:0/923916420 shutdown_connections 2024-02-19T22:09:13.699 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:13.698+0000 7fa69bc3a700 1 -- 172.21.15.52:0/923916420 wait complete. 2024-02-19T22:09:14.342 INFO:teuthology.orchestra.run.smithi052.stderr:Enabling mgr prometheus module... 2024-02-19T22:09:14.342 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi052 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphvym4zi3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp6ou7w4ul:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2024-02-19T22:09:14.921 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:14.919+0000 7fb8de872700 1 Processor -- start 2024-02-19T22:09:14.922 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:14.919+0000 7fb8de872700 1 -- start start 2024-02-19T22:09:14.922 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:14.919+0000 7fb8de872700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb8d8070390 0x7fb8d8100ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:14.922 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:14.919+0000 7fb8de872700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d80fc8e0 con 0x7fb8d8070390 2024-02-19T22:09:14.922 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:14.919+0000 7fb8d7fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb8d8070390 0x7fb8d8100ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:14.922 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:14.919+0000 7fb8d7fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb8d8070390 0x7fb8d8100ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56534/0 (socket says 172.21.15.52:56534) 2024-02-19T22:09:14.922 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:14.919+0000 7fb8d7fff700 1 -- 172.21.15.52:0/3956847084 learned_addr learned my addr 172.21.15.52:0/3956847084 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:09:14.922 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:14.919+0000 7fb8d7fff700 1 -- 172.21.15.52:0/3956847084 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8d81013e0 con 0x7fb8d8070390 2024-02-19T22:09:14.922 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:14.920+0000 7fb8d7fff700 1 --2- 172.21.15.52:0/3956847084 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb8d8070390 0x7fb8d8100ef0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fb8c800b5f0 tx=0x7fb8c8006ee0).ready entity=mon.0 client_cookie=a00784079abeb202 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:14.922 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:14.920+0000 7fb8d6ffd700 1 -- 172.21.15.52:0/3956847084 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8c800d600 con 0x7fb8d8070390 2024-02-19T22:09:14.922 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:14.920+0000 7fb8d6ffd700 1 -- 172.21.15.52:0/3956847084 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fb8c800dbf0 con 0x7fb8d8070390 2024-02-19T22:09:14.922 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:14.920+0000 7fb8d6ffd700 1 -- 172.21.15.52:0/3956847084 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8c8011cb0 con 0x7fb8d8070390 2024-02-19T22:09:14.922 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:14.920+0000 7fb8de872700 1 -- 172.21.15.52:0/3956847084 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb8d8070390 msgr2=0x7fb8d8100ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:14.922 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:14.920+0000 7fb8de872700 1 --2- 172.21.15.52:0/3956847084 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb8d8070390 0x7fb8d8100ef0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fb8c800b5f0 tx=0x7fb8c8006ee0).stop 2024-02-19T22:09:14.925 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:14.921+0000 7fb8de872700 1 -- 172.21.15.52:0/3956847084 shutdown_connections 2024-02-19T22:09:14.925 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:14.921+0000 7fb8de872700 1 --2- 172.21.15.52:0/3956847084 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb8d8070390 0x7fb8d8100ef0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:14.925 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:14.921+0000 7fb8de872700 1 -- 172.21.15.52:0/3956847084 >> 172.21.15.52:0/3956847084 conn(0x7fb8d80f6530 msgr2=0x7fb8d80f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:14.925 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:14.921+0000 7fb8de872700 1 -- 172.21.15.52:0/3956847084 shutdown_connections 2024-02-19T22:09:14.925 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:14.921+0000 7fb8de872700 1 -- 172.21.15.52:0/3956847084 wait complete. 2024-02-19T22:09:14.925 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:14.921+0000 7fb8de872700 1 Processor -- start 2024-02-19T22:09:14.925 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:14.922+0000 7fb8de872700 1 -- start start 2024-02-19T22:09:14.925 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:14.922+0000 7fb8de872700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb8d8070390 0x7fb8d810cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:14.925 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:14.922+0000 7fb8de872700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d810d270 con 0x7fb8d8070390 2024-02-19T22:09:14.925 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:14.922+0000 7fb8d7fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb8d8070390 0x7fb8d810cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:14.925 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:14.922+0000 7fb8d7fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb8d8070390 0x7fb8d810cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56544/0 (socket says 172.21.15.52:56544) 2024-02-19T22:09:14.925 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:14.922+0000 7fb8d7fff700 1 -- 172.21.15.52:0/3254472257 learned_addr learned my addr 172.21.15.52:0/3254472257 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:09:14.926 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:14.922+0000 7fb8d7fff700 1 -- 172.21.15.52:0/3254472257 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c800b200 con 0x7fb8d8070390 2024-02-19T22:09:14.926 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:14.923+0000 7fb8d7fff700 1 --2- 172.21.15.52:0/3254472257 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb8d8070390 0x7fb8d810cd80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fb8c8006800 tx=0x7fb8c8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:14.926 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:14.923+0000 7fb8d4ff9700 1 -- 172.21.15.52:0/3254472257 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8c801b570 con 0x7fb8d8070390 2024-02-19T22:09:14.926 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:14.923+0000 7fb8d4ff9700 1 -- 172.21.15.52:0/3254472257 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fb8c801bb60 con 0x7fb8d8070390 2024-02-19T22:09:14.926 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:14.923+0000 7fb8de872700 1 -- 172.21.15.52:0/3254472257 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8d810d470 con 0x7fb8d8070390 2024-02-19T22:09:14.926 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:14.923+0000 7fb8d4ff9700 1 -- 172.21.15.52:0/3254472257 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8c801a9f0 con 0x7fb8d8070390 2024-02-19T22:09:14.926 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:14.923+0000 7fb8de872700 1 -- 172.21.15.52:0/3254472257 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8d810e080 con 0x7fb8d8070390 2024-02-19T22:09:14.926 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:14.924+0000 7fb8d4ff9700 1 -- 172.21.15.52:0/3254472257 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fb8c800f070 con 0x7fb8d8070390 2024-02-19T22:09:14.927 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:14.924+0000 7fb8d4ff9700 1 --2- 172.21.15.52:0/3254472257 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7fb8c402fe10 0x7fb8c40322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:14.927 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:14.924+0000 7fb8de872700 1 -- 172.21.15.52:0/3254472257 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8d8059100 con 0x7fb8d8070390 2024-02-19T22:09:14.927 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:14.924+0000 7fb8d4ff9700 1 -- 172.21.15.52:0/3254472257 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fb8c8041450 con 0x7fb8d8070390 2024-02-19T22:09:14.927 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:14.924+0000 7fb8d77fe700 1 --2- 172.21.15.52:0/3254472257 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7fb8c402fe10 0x7fb8c40322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:14.927 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:14.925+0000 7fb8d77fe700 1 --2- 172.21.15.52:0/3254472257 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7fb8c402fe10 0x7fb8c40322d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fb8cc0096a0 tx=0x7fb8cc006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:14.929 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:14.929+0000 7fb8d4ff9700 1 -- 172.21.15.52:0/3254472257 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb8c8016020 con 0x7fb8d8070390 2024-02-19T22:09:15.091 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:14 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:13.693913+0000 mgr.smithi052.ghqrtk (mgr.14114) 14 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:09:15.092 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:14 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cephadm 2024-02-19T22:09:13.694539+0000 mgr.smithi052.ghqrtk (mgr.14114) 15 : cephadm [INF] Saving service crash spec with placement * 2024-02-19T22:09:15.092 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:14 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:13.696583+0000 mon.smithi052 (mon.0) 58 : audit [INF] from='mgr.14114 172.21.15.52:0/3613687413' entity='mgr.smithi052.ghqrtk' 2024-02-19T22:09:15.127 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:15.126+0000 7fb8de872700 1 -- 172.21.15.52:0/3254472257 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7fb8d810eeb0 con 0x7fb8d8070390 2024-02-19T22:09:15.700 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:15.699+0000 7fb8d4ff9700 1 -- 172.21.15.52:0/3254472257 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7fb8c801ab50 con 0x7fb8d8070390 2024-02-19T22:09:15.701 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:15.700+0000 7fb8d4ff9700 1 -- 172.21.15.52:0/3254472257 <== mon.0 v2:172.21.15.52:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v9) v1 ==== 89+0+0 (secure 0 0 0) 0x7fb8c80249d0 con 0x7fb8d8070390 2024-02-19T22:09:15.705 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:15.703+0000 7fb8de872700 1 -- 172.21.15.52:0/3254472257 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7fb8c402fe10 msgr2=0x7fb8c40322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:15.705 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:15.703+0000 7fb8de872700 1 --2- 172.21.15.52:0/3254472257 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7fb8c402fe10 0x7fb8c40322d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fb8cc0096a0 tx=0x7fb8cc006b40).stop 2024-02-19T22:09:15.705 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:15.703+0000 7fb8de872700 1 -- 172.21.15.52:0/3254472257 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb8d8070390 msgr2=0x7fb8d810cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:15.705 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:15.703+0000 7fb8de872700 1 --2- 172.21.15.52:0/3254472257 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb8d8070390 0x7fb8d810cd80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fb8c8006800 tx=0x7fb8c8006ee0).stop 2024-02-19T22:09:15.705 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:15.703+0000 7fb8de872700 1 -- 172.21.15.52:0/3254472257 shutdown_connections 2024-02-19T22:09:15.705 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:15.703+0000 7fb8de872700 1 --2- 172.21.15.52:0/3254472257 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7fb8c402fe10 0x7fb8c40322d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:15.705 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:15.703+0000 7fb8de872700 1 --2- 172.21.15.52:0/3254472257 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb8d8070390 0x7fb8d810cd80 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:15.706 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:15.703+0000 7fb8de872700 1 -- 172.21.15.52:0/3254472257 >> 172.21.15.52:0/3254472257 conn(0x7fb8d80f6530 msgr2=0x7fb8d80f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:15.706 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:15.704+0000 7fb8de872700 1 -- 172.21.15.52:0/3254472257 shutdown_connections 2024-02-19T22:09:15.706 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:15.704+0000 7fb8de872700 1 -- 172.21.15.52:0/3254472257 wait complete. 2024-02-19T22:09:16.091 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:15 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:15.128168+0000 mon.smithi052 (mon.0) 59 : audit [INF] from='client.? 172.21.15.52:0/3254472257' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-19T22:09:16.632 INFO:teuthology.orchestra.run.smithi052.stderr:Deploying prometheus service with default placement... 2024-02-19T22:09:16.632 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi052 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphvym4zi3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp6ou7w4ul:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2024-02-19T22:09:16.972 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:16 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:15.701608+0000 mon.smithi052 (mon.0) 60 : audit [INF] from='client.? 172.21.15.52:0/3254472257' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-19T22:09:16.972 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:16 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:09:15.701683+0000 mon.smithi052 (mon.0) 61 : cluster [DBG] mgrmap e9: smithi052.ghqrtk(active, since 14s) 2024-02-19T22:09:17.412 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:17.409+0000 7f311ca01700 1 Processor -- start 2024-02-19T22:09:17.413 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:17.409+0000 7f311ca01700 1 -- start start 2024-02-19T22:09:17.413 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:17.409+0000 7f311ca01700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f31180703b0 0x7f3118100d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:17.413 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:17.409+0000 7f311ca01700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31180fc700 con 0x7f31180703b0 2024-02-19T22:09:17.413 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:17.409+0000 7f3116d9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f31180703b0 0x7f3118100d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:17.413 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:17.409+0000 7f3116d9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f31180703b0 0x7f3118100d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56564/0 (socket says 172.21.15.52:56564) 2024-02-19T22:09:17.413 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:17.409+0000 7f3116d9d700 1 -- 172.21.15.52:0/1523603279 learned_addr learned my addr 172.21.15.52:0/1523603279 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:09:17.413 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:17.410+0000 7f3116d9d700 1 -- 172.21.15.52:0/1523603279 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3118101200 con 0x7f31180703b0 2024-02-19T22:09:17.413 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:17.410+0000 7f3116d9d700 1 --2- 172.21.15.52:0/1523603279 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f31180703b0 0x7f3118100d10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f310800b5f0 tx=0x7f3108006ee0).ready entity=mon.0 client_cookie=ec1807e78b0183ba server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:17.413 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:17.410+0000 7f3115d9b700 1 -- 172.21.15.52:0/1523603279 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f310800d600 con 0x7f31180703b0 2024-02-19T22:09:17.413 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:17.410+0000 7f3115d9b700 1 -- 172.21.15.52:0/1523603279 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f310800dbf0 con 0x7f31180703b0 2024-02-19T22:09:17.413 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:17.410+0000 7f3115d9b700 1 -- 172.21.15.52:0/1523603279 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3108011cb0 con 0x7f31180703b0 2024-02-19T22:09:17.413 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:17.410+0000 7f311ca01700 1 -- 172.21.15.52:0/1523603279 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f31180703b0 msgr2=0x7f3118100d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:17.413 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:17.411+0000 7f311ca01700 1 --2- 172.21.15.52:0/1523603279 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f31180703b0 0x7f3118100d10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f310800b5f0 tx=0x7f3108006ee0).stop 2024-02-19T22:09:17.413 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:17.411+0000 7f311ca01700 1 -- 172.21.15.52:0/1523603279 shutdown_connections 2024-02-19T22:09:17.414 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:17.411+0000 7f311ca01700 1 --2- 172.21.15.52:0/1523603279 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f31180703b0 0x7f3118100d10 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:17.414 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:17.411+0000 7f311ca01700 1 -- 172.21.15.52:0/1523603279 >> 172.21.15.52:0/1523603279 conn(0x7f31180f6390 msgr2=0x7f31180f87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:17.414 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:17.411+0000 7f311ca01700 1 -- 172.21.15.52:0/1523603279 shutdown_connections 2024-02-19T22:09:17.414 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:17.411+0000 7f311ca01700 1 -- 172.21.15.52:0/1523603279 wait complete. 2024-02-19T22:09:17.415 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:17.411+0000 7f311ca01700 1 Processor -- start 2024-02-19T22:09:17.415 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:17.411+0000 7f311ca01700 1 -- start start 2024-02-19T22:09:17.415 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:17.412+0000 7f311ca01700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f31180703b0 0x7f311810cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:17.415 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:17.412+0000 7f311ca01700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311810d0b0 con 0x7f31180703b0 2024-02-19T22:09:17.415 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:17.412+0000 7f3116d9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f31180703b0 0x7f311810cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:17.415 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:17.412+0000 7f3116d9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f31180703b0 0x7f311810cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56572/0 (socket says 172.21.15.52:56572) 2024-02-19T22:09:17.415 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:17.412+0000 7f3116d9d700 1 -- 172.21.15.52:0/4230866805 learned_addr learned my addr 172.21.15.52:0/4230866805 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:09:17.415 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:17.412+0000 7f3116d9d700 1 -- 172.21.15.52:0/4230866805 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f310800b200 con 0x7f31180703b0 2024-02-19T22:09:17.415 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:17.412+0000 7f3116d9d700 1 --2- 172.21.15.52:0/4230866805 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f31180703b0 0x7f311810cbc0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f310800df90 tx=0x7f3108006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:17.415 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:17.412+0000 7f30ff7fe700 1 -- 172.21.15.52:0/4230866805 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f310801b410 con 0x7f31180703b0 2024-02-19T22:09:17.416 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:17.412+0000 7f30ff7fe700 1 -- 172.21.15.52:0/4230866805 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f310801ba00 con 0x7f31180703b0 2024-02-19T22:09:17.416 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:17.412+0000 7f311ca01700 1 -- 172.21.15.52:0/4230866805 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f311810d2b0 con 0x7f31180703b0 2024-02-19T22:09:17.416 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:17.412+0000 7f30ff7fe700 1 -- 172.21.15.52:0/4230866805 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f310801aa30 con 0x7f31180703b0 2024-02-19T22:09:17.416 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:17.412+0000 7f311ca01700 1 -- 172.21.15.52:0/4230866805 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f311810d740 con 0x7f31180703b0 2024-02-19T22:09:17.416 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:17.413+0000 7f30ff7fe700 1 -- 172.21.15.52:0/4230866805 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f310800f070 con 0x7f31180703b0 2024-02-19T22:09:17.416 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:17.413+0000 7f311ca01700 1 -- 172.21.15.52:0/4230866805 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3118045ad0 con 0x7f31180703b0 2024-02-19T22:09:17.416 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:17.414+0000 7f30ff7fe700 1 --2- 172.21.15.52:0/4230866805 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7f310402fe70 0x7f3104032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:17.416 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:17.414+0000 7f30ff7fe700 1 -- 172.21.15.52:0/4230866805 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f3108041210 con 0x7f31180703b0 2024-02-19T22:09:17.416 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:17.414+0000 7f311659c700 1 -- 172.21.15.52:0/4230866805 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7f310402fe70 msgr2=0x7f3104032330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/892530721 2024-02-19T22:09:17.416 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:17.414+0000 7f311659c700 1 --2- 172.21.15.52:0/4230866805 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7f310402fe70 0x7f3104032330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-19T22:09:17.419 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:17.418+0000 7f30ff7fe700 1 -- 172.21.15.52:0/4230866805 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3108022d20 con 0x7f31180703b0 2024-02-19T22:09:17.610 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:17.608+0000 7f311ca01700 1 -- 172.21.15.52:0/4230866805 --> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f311810e060 con 0x7f310402fe70 2024-02-19T22:09:17.616 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:17.615+0000 7f311659c700 1 -- 172.21.15.52:0/4230866805 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7f310402fe70 msgr2=0x7f3104032330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/892530721 2024-02-19T22:09:17.616 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:17.615+0000 7f311659c700 1 --2- 172.21.15.52:0/4230866805 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7f310402fe70 0x7f3104032330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-19T22:09:18.017 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:18.016+0000 7f311659c700 1 -- 172.21.15.52:0/4230866805 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7f310402fe70 msgr2=0x7f3104032330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/892530721 2024-02-19T22:09:18.017 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:18.016+0000 7f311659c700 1 --2- 172.21.15.52:0/4230866805 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7f310402fe70 0x7f3104032330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-19T22:09:18.818 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:18.818+0000 7f311659c700 1 -- 172.21.15.52:0/4230866805 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7f310402fe70 msgr2=0x7f3104032330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/892530721 2024-02-19T22:09:18.818 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:18.818+0000 7f311659c700 1 --2- 172.21.15.52:0/4230866805 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7f310402fe70 0x7f3104032330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-02-19T22:09:20.419 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:20.419+0000 7f311659c700 1 -- 172.21.15.52:0/4230866805 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7f310402fe70 msgr2=0x7f3104032330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/892530721 2024-02-19T22:09:20.419 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:20.419+0000 7f311659c700 1 --2- 172.21.15.52:0/4230866805 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7f310402fe70 0x7f3104032330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-02-19T22:09:20.969 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:20.969+0000 7f30ff7fe700 1 -- 172.21.15.52:0/4230866805 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mgrmap(e 10) v1 ==== 35984+0+0 (secure 0 0 0) 0x7f3108025980 con 0x7f31180703b0 2024-02-19T22:09:20.969 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:20.969+0000 7f30ff7fe700 1 -- 172.21.15.52:0/4230866805 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7f310402fe70 msgr2=0x7f3104032330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T22:09:20.969 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:20.969+0000 7f30ff7fe700 1 --2- 172.21.15.52:0/4230866805 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7f310402fe70 0x7f3104032330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T22:09:21.972 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:21.970+0000 7f30ff7fe700 1 -- 172.21.15.52:0/4230866805 <== mon.0 v2:172.21.15.52:3300/0 8 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f310803c930 con 0x7f31180703b0 2024-02-19T22:09:21.972 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:21.970+0000 7f30ff7fe700 1 --2- 172.21.15.52:0/4230866805 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f3104033ce0 0x7f31040360d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:21.972 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:21.970+0000 7f30ff7fe700 1 -- 172.21.15.52:0/4230866805 --> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f311810e060 con 0x7f3104033ce0 2024-02-19T22:09:21.972 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:21.970+0000 7f311659c700 1 --2- 172.21.15.52:0/4230866805 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f3104033ce0 0x7f31040360d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:21.972 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:21.971+0000 7f311659c700 1 --2- 172.21.15.52:0/4230866805 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f3104033ce0 0x7f31040360d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f310c0036d0 tx=0x7f310c006cb0).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:21.976 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:21.976+0000 7f30ff7fe700 1 -- 172.21.15.52:0/4230866805 <== mgr.14142 v2:172.21.15.52:6800/2071220111 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f311810e060 con 0x7f3104033ce0 2024-02-19T22:09:21.976 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-02-19T22:09:21.980 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:21.978+0000 7f311ca01700 1 -- 172.21.15.52:0/4230866805 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f3104033ce0 msgr2=0x7f31040360d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:21.980 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:21.978+0000 7f311ca01700 1 --2- 172.21.15.52:0/4230866805 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f3104033ce0 0x7f31040360d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f310c0036d0 tx=0x7f310c006cb0).stop 2024-02-19T22:09:21.980 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:21.978+0000 7f311ca01700 1 -- 172.21.15.52:0/4230866805 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f31180703b0 msgr2=0x7f311810cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:21.980 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:21.978+0000 7f311ca01700 1 --2- 172.21.15.52:0/4230866805 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f31180703b0 0x7f311810cbc0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f310800df90 tx=0x7f3108006ee0).stop 2024-02-19T22:09:21.980 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:21.978+0000 7f311ca01700 1 -- 172.21.15.52:0/4230866805 shutdown_connections 2024-02-19T22:09:21.980 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:21.978+0000 7f311ca01700 1 --2- 172.21.15.52:0/4230866805 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f3104033ce0 0x7f31040360d0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:21.980 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:21.978+0000 7f311ca01700 1 --2- 172.21.15.52:0/4230866805 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f31180703b0 0x7f311810cbc0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:21.981 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:21.978+0000 7f311ca01700 1 --2- 172.21.15.52:0/4230866805 >> [v2:172.21.15.52:6800/892530721,v1:172.21.15.52:6801/892530721] conn(0x7f310402fe70 0x7f3104032330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T22:09:21.981 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:21.978+0000 7f311ca01700 1 -- 172.21.15.52:0/4230866805 >> 172.21.15.52:0/4230866805 conn(0x7f31180f6390 msgr2=0x7f31180f7e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:21.981 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:21.978+0000 7f311ca01700 1 -- 172.21.15.52:0/4230866805 shutdown_connections 2024-02-19T22:09:21.981 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:21.978+0000 7f311ca01700 1 -- 172.21.15.52:0/4230866805 wait complete. 2024-02-19T22:09:22.293 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:21 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:09:20.917124+0000 mon.smithi052 (mon.0) 62 : cluster [INF] Active manager daemon smithi052.ghqrtk restarted 2024-02-19T22:09:22.294 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:21 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:09:20.917666+0000 mon.smithi052 (mon.0) 63 : cluster [INF] Activating manager daemon smithi052.ghqrtk 2024-02-19T22:09:22.294 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:21 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:09:20.919214+0000 mon.smithi052 (mon.0) 64 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-02-19T22:09:22.294 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:21 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:09:20.970431+0000 mon.smithi052 (mon.0) 65 : cluster [DBG] mgrmap e10: smithi052.ghqrtk(active, starting, since 0.0528859s) 2024-02-19T22:09:22.294 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:21 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:20.971505+0000 mon.smithi052 (mon.0) 66 : audit [DBG] from='mgr.14142 172.21.15.52:0/284471578' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix": "mon metadata", "id": "smithi052"}]: dispatch 2024-02-19T22:09:22.294 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:21 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:20.972045+0000 mon.smithi052 (mon.0) 67 : audit [DBG] from='mgr.14142 172.21.15.52:0/284471578' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix": "mgr metadata", "who": "smithi052.ghqrtk", "id": "smithi052.ghqrtk"}]: dispatch 2024-02-19T22:09:22.294 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:21 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:20.972753+0000 mon.smithi052 (mon.0) 68 : audit [DBG] from='mgr.14142 172.21.15.52:0/284471578' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T22:09:22.294 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:21 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:20.972976+0000 mon.smithi052 (mon.0) 69 : audit [DBG] from='mgr.14142 172.21.15.52:0/284471578' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T22:09:22.294 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:21 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:20.973183+0000 mon.smithi052 (mon.0) 70 : audit [DBG] from='mgr.14142 172.21.15.52:0/284471578' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T22:09:22.294 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:21 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:09:20.975141+0000 mon.smithi052 (mon.0) 71 : cluster [INF] Manager daemon smithi052.ghqrtk is now available 2024-02-19T22:09:22.294 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:21 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:21.079080+0000 mon.smithi052 (mon.0) 72 : audit [DBG] from='mgr.14142 172.21.15.52:0/284471578' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T22:09:22.294 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:21 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:21.080651+0000 mon.smithi052 (mon.0) 73 : audit [DBG] from='mgr.14142 172.21.15.52:0/284471578' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T22:09:22.294 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:21 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:21.095744+0000 mon.smithi052 (mon.0) 74 : audit [INF] from='mgr.14142 172.21.15.52:0/284471578' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi052.ghqrtk/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:09:22.295 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:21 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:21.104008+0000 mon.smithi052 (mon.0) 75 : audit [INF] from='mgr.14142 172.21.15.52:0/284471578' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi052.ghqrtk/trash_purge_schedule"}]: dispatch 2024-02-19T22:09:22.591 INFO:teuthology.orchestra.run.smithi052.stderr:Deploying grafana service with default placement... 2024-02-19T22:09:22.591 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi052 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphvym4zi3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp6ou7w4ul:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2024-02-19T22:09:23.129 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.124+0000 7f41c5d91700 1 Processor -- start 2024-02-19T22:09:23.129 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.124+0000 7f41c5d91700 1 -- start start 2024-02-19T22:09:23.129 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.124+0000 7f41c5d91700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f41c00703d0 0x7f41c0100eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:23.129 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.124+0000 7f41c5d91700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c00fc8a0 con 0x7f41c00703d0 2024-02-19T22:09:23.129 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.125+0000 7f41bf7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f41c00703d0 0x7f41c0100eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:23.129 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.125+0000 7f41bf7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f41c00703d0 0x7f41c0100eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56642/0 (socket says 172.21.15.52:56642) 2024-02-19T22:09:23.129 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.125+0000 7f41bf7fe700 1 -- 172.21.15.52:0/473802289 learned_addr learned my addr 172.21.15.52:0/473802289 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:09:23.129 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.125+0000 7f41bf7fe700 1 -- 172.21.15.52:0/473802289 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41c01013a0 con 0x7f41c00703d0 2024-02-19T22:09:23.129 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.125+0000 7f41bf7fe700 1 --2- 172.21.15.52:0/473802289 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f41c00703d0 0x7f41c0100eb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f41b000b5f0 tx=0x7f41b0006ee0).ready entity=mon.0 client_cookie=c21d8603b81fdf1d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:23.129 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.125+0000 7f41be7fc700 1 -- 172.21.15.52:0/473802289 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f41b000d600 con 0x7f41c00703d0 2024-02-19T22:09:23.129 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.125+0000 7f41be7fc700 1 -- 172.21.15.52:0/473802289 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f41b000dbf0 con 0x7f41c00703d0 2024-02-19T22:09:23.129 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.125+0000 7f41be7fc700 1 -- 172.21.15.52:0/473802289 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f41b0011cb0 con 0x7f41c00703d0 2024-02-19T22:09:23.130 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.126+0000 7f41c5d91700 1 -- 172.21.15.52:0/473802289 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f41c00703d0 msgr2=0x7f41c0100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:23.130 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.126+0000 7f41c5d91700 1 --2- 172.21.15.52:0/473802289 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f41c00703d0 0x7f41c0100eb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f41b000b5f0 tx=0x7f41b0006ee0).stop 2024-02-19T22:09:23.130 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.126+0000 7f41c5d91700 1 -- 172.21.15.52:0/473802289 shutdown_connections 2024-02-19T22:09:23.130 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.126+0000 7f41c5d91700 1 --2- 172.21.15.52:0/473802289 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f41c00703d0 0x7f41c0100eb0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:23.130 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.126+0000 7f41c5d91700 1 -- 172.21.15.52:0/473802289 >> 172.21.15.52:0/473802289 conn(0x7f41c00f6530 msgr2=0x7f41c00f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:23.130 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.126+0000 7f41c5d91700 1 -- 172.21.15.52:0/473802289 shutdown_connections 2024-02-19T22:09:23.130 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.126+0000 7f41c5d91700 1 -- 172.21.15.52:0/473802289 wait complete. 2024-02-19T22:09:23.130 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.127+0000 7f41c5d91700 1 Processor -- start 2024-02-19T22:09:23.130 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.127+0000 7f41c5d91700 1 -- start start 2024-02-19T22:09:23.130 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.127+0000 7f41c5d91700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f41c00703d0 0x7f41c010cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:23.130 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.127+0000 7f41c5d91700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c010d2a0 con 0x7f41c00703d0 2024-02-19T22:09:23.130 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.127+0000 7f41bf7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f41c00703d0 0x7f41c010cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:23.130 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.127+0000 7f41bf7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f41c00703d0 0x7f41c010cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56654/0 (socket says 172.21.15.52:56654) 2024-02-19T22:09:23.130 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.127+0000 7f41bf7fe700 1 -- 172.21.15.52:0/405714243 learned_addr learned my addr 172.21.15.52:0/405714243 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:09:23.131 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.128+0000 7f41bf7fe700 1 -- 172.21.15.52:0/405714243 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b000b200 con 0x7f41c00703d0 2024-02-19T22:09:23.131 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.128+0000 7f41bf7fe700 1 --2- 172.21.15.52:0/405714243 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f41c00703d0 0x7f41c010cdb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f41b0006a20 tx=0x7f41b0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:23.131 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.128+0000 7f41b7fff700 1 -- 172.21.15.52:0/405714243 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f41b001b570 con 0x7f41c00703d0 2024-02-19T22:09:23.131 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.128+0000 7f41c5d91700 1 -- 172.21.15.52:0/405714243 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41c010d4a0 con 0x7f41c00703d0 2024-02-19T22:09:23.131 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.128+0000 7f41b7fff700 1 -- 172.21.15.52:0/405714243 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f41b001bb60 con 0x7f41c00703d0 2024-02-19T22:09:23.131 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.128+0000 7f41b7fff700 1 -- 172.21.15.52:0/405714243 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f41b001aa20 con 0x7f41c00703d0 2024-02-19T22:09:23.131 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.128+0000 7f41c5d91700 1 -- 172.21.15.52:0/405714243 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41c010e0e0 con 0x7f41c00703d0 2024-02-19T22:09:23.132 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.129+0000 7f41c5d91700 1 -- 172.21.15.52:0/405714243 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41c010d930 con 0x7f41c00703d0 2024-02-19T22:09:23.132 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.129+0000 7f41b7fff700 1 -- 172.21.15.52:0/405714243 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f41b000f070 con 0x7f41c00703d0 2024-02-19T22:09:23.132 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.129+0000 7f41b7fff700 1 --2- 172.21.15.52:0/405714243 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f41a002ff20 0x7f41a00323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:23.132 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.129+0000 7f41b7fff700 1 -- 172.21.15.52:0/405714243 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f41b00414d0 con 0x7f41c00703d0 2024-02-19T22:09:23.132 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.129+0000 7f41beffd700 1 --2- 172.21.15.52:0/405714243 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f41a002ff20 0x7f41a00323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:23.132 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.130+0000 7f41beffd700 1 --2- 172.21.15.52:0/405714243 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f41a002ff20 0x7f41a00323e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f41a80096a0 tx=0x7f41a8006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:23.133 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.133+0000 7f41b7fff700 1 -- 172.21.15.52:0/405714243 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f41b001a370 con 0x7f41c00703d0 2024-02-19T22:09:23.318 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.318+0000 7f41c5d91700 1 -- 172.21.15.52:0/405714243 --> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f41c010e3e0 con 0x7f41a002ff20 2024-02-19T22:09:23.321 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.321+0000 7f41b7fff700 1 -- 172.21.15.52:0/405714243 <== mgr.14142 v2:172.21.15.52:6800/2071220111 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f41c010e3e0 con 0x7f41a002ff20 2024-02-19T22:09:23.321 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-02-19T22:09:23.324 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.323+0000 7f41c5d91700 1 -- 172.21.15.52:0/405714243 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f41a002ff20 msgr2=0x7f41a00323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:23.324 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.323+0000 7f41c5d91700 1 --2- 172.21.15.52:0/405714243 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f41a002ff20 0x7f41a00323e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f41a80096a0 tx=0x7f41a8006b40).stop 2024-02-19T22:09:23.324 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.323+0000 7f41c5d91700 1 -- 172.21.15.52:0/405714243 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f41c00703d0 msgr2=0x7f41c010cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:23.324 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.323+0000 7f41c5d91700 1 --2- 172.21.15.52:0/405714243 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f41c00703d0 0x7f41c010cdb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f41b0006a20 tx=0x7f41b0006ee0).stop 2024-02-19T22:09:23.324 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.323+0000 7f41c5d91700 1 -- 172.21.15.52:0/405714243 shutdown_connections 2024-02-19T22:09:23.324 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.323+0000 7f41c5d91700 1 --2- 172.21.15.52:0/405714243 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f41a002ff20 0x7f41a00323e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:23.324 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.323+0000 7f41c5d91700 1 --2- 172.21.15.52:0/405714243 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f41c00703d0 0x7f41c010cdb0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:23.324 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.323+0000 7f41c5d91700 1 -- 172.21.15.52:0/405714243 >> 172.21.15.52:0/405714243 conn(0x7f41c00f6530 msgr2=0x7f41c00f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:23.324 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.323+0000 7f41c5d91700 1 -- 172.21.15.52:0/405714243 shutdown_connections 2024-02-19T22:09:23.324 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:23.323+0000 7f41c5d91700 1 -- 172.21.15.52:0/405714243 wait complete. 2024-02-19T22:09:23.341 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:22 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:09:21.972335+0000 mon.smithi052 (mon.0) 76 : cluster [DBG] mgrmap e11: smithi052.ghqrtk(active, since 1.05479s) 2024-02-19T22:09:23.341 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:22 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cephadm 2024-02-19T22:09:21.973642+0000 mgr.smithi052.ghqrtk (mgr.14142) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2024-02-19T22:09:23.342 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:22 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:21.975795+0000 mon.smithi052 (mon.0) 77 : audit [INF] from='mgr.14142 172.21.15.52:0/284471578' entity='mgr.smithi052.ghqrtk' 2024-02-19T22:09:23.989 INFO:teuthology.orchestra.run.smithi052.stderr:Deploying node-exporter service with default placement... 2024-02-19T22:09:23.990 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi052 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphvym4zi3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp6ou7w4ul:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2024-02-19T22:09:24.091 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:23 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:09:22.976774+0000 mon.smithi052 (mon.0) 78 : cluster [DBG] mgrmap e12: smithi052.ghqrtk(active, since 2s) 2024-02-19T22:09:24.092 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:23 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:23.318960+0000 mgr.smithi052.ghqrtk (mgr.14142) 3 : audit [DBG] from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:09:24.092 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:23 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cephadm 2024-02-19T22:09:23.319449+0000 mgr.smithi052.ghqrtk (mgr.14142) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2024-02-19T22:09:24.092 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:23 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:23.321617+0000 mon.smithi052 (mon.0) 79 : audit [INF] from='mgr.14142 172.21.15.52:0/284471578' entity='mgr.smithi052.ghqrtk' 2024-02-19T22:09:24.510 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.493+0000 7f9681f0b700 1 Processor -- start 2024-02-19T22:09:24.510 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.493+0000 7f9681f0b700 1 -- start start 2024-02-19T22:09:24.511 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.493+0000 7f9681f0b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f967c0703d0 0x7f967c100eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:24.511 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.493+0000 7f9681f0b700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f967c0fc8a0 con 0x7f967c0703d0 2024-02-19T22:09:24.511 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.494+0000 7f967b7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f967c0703d0 0x7f967c100eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:24.511 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.494+0000 7f967b7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f967c0703d0 0x7f967c100eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:52964/0 (socket says 172.21.15.52:52964) 2024-02-19T22:09:24.511 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.494+0000 7f967b7fe700 1 -- 172.21.15.52:0/568927159 learned_addr learned my addr 172.21.15.52:0/568927159 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:09:24.511 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.494+0000 7f967b7fe700 1 -- 172.21.15.52:0/568927159 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f967c1013a0 con 0x7f967c0703d0 2024-02-19T22:09:24.511 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.494+0000 7f967b7fe700 1 --2- 172.21.15.52:0/568927159 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f967c0703d0 0x7f967c100eb0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f966c00b5f0 tx=0x7f966c006ee0).ready entity=mon.0 client_cookie=442c245acb5e1e2f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:24.511 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.494+0000 7f967a7fc700 1 -- 172.21.15.52:0/568927159 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f966c00d600 con 0x7f967c0703d0 2024-02-19T22:09:24.511 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.494+0000 7f967a7fc700 1 -- 172.21.15.52:0/568927159 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f966c00dbf0 con 0x7f967c0703d0 2024-02-19T22:09:24.511 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.494+0000 7f967a7fc700 1 -- 172.21.15.52:0/568927159 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f966c011cb0 con 0x7f967c0703d0 2024-02-19T22:09:24.511 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.495+0000 7f9681f0b700 1 -- 172.21.15.52:0/568927159 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f967c0703d0 msgr2=0x7f967c100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:24.511 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.495+0000 7f9681f0b700 1 --2- 172.21.15.52:0/568927159 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f967c0703d0 0x7f967c100eb0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f966c00b5f0 tx=0x7f966c006ee0).stop 2024-02-19T22:09:24.511 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.495+0000 7f9681f0b700 1 -- 172.21.15.52:0/568927159 shutdown_connections 2024-02-19T22:09:24.512 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.495+0000 7f9681f0b700 1 --2- 172.21.15.52:0/568927159 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f967c0703d0 0x7f967c100eb0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:24.512 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.495+0000 7f9681f0b700 1 -- 172.21.15.52:0/568927159 >> 172.21.15.52:0/568927159 conn(0x7f967c0f6530 msgr2=0x7f967c0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:24.512 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.495+0000 7f9681f0b700 1 -- 172.21.15.52:0/568927159 shutdown_connections 2024-02-19T22:09:24.512 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.495+0000 7f9681f0b700 1 -- 172.21.15.52:0/568927159 wait complete. 2024-02-19T22:09:24.512 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.495+0000 7f9681f0b700 1 Processor -- start 2024-02-19T22:09:24.512 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.495+0000 7f9681f0b700 1 -- start start 2024-02-19T22:09:24.512 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.496+0000 7f9681f0b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f967c0703d0 0x7f967c10cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:24.512 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.496+0000 7f9681f0b700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f967c10d2a0 con 0x7f967c0703d0 2024-02-19T22:09:24.512 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.496+0000 7f967b7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f967c0703d0 0x7f967c10cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:24.512 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.496+0000 7f967b7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f967c0703d0 0x7f967c10cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:52978/0 (socket says 172.21.15.52:52978) 2024-02-19T22:09:24.512 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.496+0000 7f967b7fe700 1 -- 172.21.15.52:0/4230631595 learned_addr learned my addr 172.21.15.52:0/4230631595 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:09:24.512 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.496+0000 7f967b7fe700 1 -- 172.21.15.52:0/4230631595 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f966c00b200 con 0x7f967c0703d0 2024-02-19T22:09:24.512 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.496+0000 7f967b7fe700 1 --2- 172.21.15.52:0/4230631595 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f967c0703d0 0x7f967c10cdb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f966c006a20 tx=0x7f966c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:24.512 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.496+0000 7f9673fff700 1 -- 172.21.15.52:0/4230631595 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f966c01b570 con 0x7f967c0703d0 2024-02-19T22:09:24.513 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.496+0000 7f9673fff700 1 -- 172.21.15.52:0/4230631595 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f966c01bb60 con 0x7f967c0703d0 2024-02-19T22:09:24.513 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.496+0000 7f9681f0b700 1 -- 172.21.15.52:0/4230631595 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f967c10d4a0 con 0x7f967c0703d0 2024-02-19T22:09:24.513 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.496+0000 7f9673fff700 1 -- 172.21.15.52:0/4230631595 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f966c01aa20 con 0x7f967c0703d0 2024-02-19T22:09:24.513 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.496+0000 7f9681f0b700 1 -- 172.21.15.52:0/4230631595 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f967c10e0e0 con 0x7f967c0703d0 2024-02-19T22:09:24.513 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.497+0000 7f9673fff700 1 -- 172.21.15.52:0/4230631595 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f966c00f070 con 0x7f967c0703d0 2024-02-19T22:09:24.513 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.497+0000 7f9673fff700 1 --2- 172.21.15.52:0/4230631595 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f965c02ffc0 0x7f965c032480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:24.513 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.497+0000 7f9681f0b700 1 -- 172.21.15.52:0/4230631595 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f967c10d930 con 0x7f967c0703d0 2024-02-19T22:09:24.513 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.497+0000 7f9673fff700 1 -- 172.21.15.52:0/4230631595 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f966c0414c0 con 0x7f967c0703d0 2024-02-19T22:09:24.513 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.497+0000 7f967affd700 1 --2- 172.21.15.52:0/4230631595 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f965c02ffc0 0x7f965c032480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:24.513 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.498+0000 7f967affd700 1 --2- 172.21.15.52:0/4230631595 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f965c02ffc0 0x7f965c032480 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f96640096a0 tx=0x7f9664006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:24.513 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.502+0000 7f9673fff700 1 -- 172.21.15.52:0/4230631595 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f966c01a370 con 0x7f967c0703d0 2024-02-19T22:09:24.689 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.689+0000 7f9681f0b700 1 -- 172.21.15.52:0/4230631595 --> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f967c10e320 con 0x7f965c02ffc0 2024-02-19T22:09:24.693 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.693+0000 7f9673fff700 1 -- 172.21.15.52:0/4230631595 <== mgr.14142 v2:172.21.15.52:6800/2071220111 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f967c10e320 con 0x7f965c02ffc0 2024-02-19T22:09:24.693 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-02-19T22:09:24.696 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.695+0000 7f9681f0b700 1 -- 172.21.15.52:0/4230631595 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f965c02ffc0 msgr2=0x7f965c032480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:24.696 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.695+0000 7f9681f0b700 1 --2- 172.21.15.52:0/4230631595 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f965c02ffc0 0x7f965c032480 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f96640096a0 tx=0x7f9664006b40).stop 2024-02-19T22:09:24.696 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.695+0000 7f9681f0b700 1 -- 172.21.15.52:0/4230631595 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f967c0703d0 msgr2=0x7f967c10cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:24.696 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.695+0000 7f9681f0b700 1 --2- 172.21.15.52:0/4230631595 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f967c0703d0 0x7f967c10cdb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f966c006a20 tx=0x7f966c006ee0).stop 2024-02-19T22:09:24.696 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.695+0000 7f9681f0b700 1 -- 172.21.15.52:0/4230631595 shutdown_connections 2024-02-19T22:09:24.696 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.695+0000 7f9681f0b700 1 --2- 172.21.15.52:0/4230631595 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f965c02ffc0 0x7f965c032480 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:24.696 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.695+0000 7f9681f0b700 1 --2- 172.21.15.52:0/4230631595 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f967c0703d0 0x7f967c10cdb0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:24.696 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.695+0000 7f9681f0b700 1 -- 172.21.15.52:0/4230631595 >> 172.21.15.52:0/4230631595 conn(0x7f967c0f6530 msgr2=0x7f967c0f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:24.696 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.695+0000 7f9681f0b700 1 -- 172.21.15.52:0/4230631595 shutdown_connections 2024-02-19T22:09:24.696 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:24.695+0000 7f9681f0b700 1 -- 172.21.15.52:0/4230631595 wait complete. 2024-02-19T22:09:25.379 INFO:teuthology.orchestra.run.smithi052.stderr:Deploying alertmanager service with default placement... 2024-02-19T22:09:25.379 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi052 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphvym4zi3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp6ou7w4ul:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2024-02-19T22:09:25.883 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:25.862+0000 7f6f83d54700 1 Processor -- start 2024-02-19T22:09:25.884 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:25.863+0000 7f6f83d54700 1 -- start start 2024-02-19T22:09:25.884 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:25.863+0000 7f6f83d54700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6f7c0703f0 0x7f6f7c100e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:25.884 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:25.863+0000 7f6f83d54700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7c0fc880 con 0x7f6f7c0703f0 2024-02-19T22:09:25.884 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:25.863+0000 7f6f81af0700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6f7c0703f0 0x7f6f7c100e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:25.884 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:25.863+0000 7f6f81af0700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6f7c0703f0 0x7f6f7c100e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:52984/0 (socket says 172.21.15.52:52984) 2024-02-19T22:09:25.884 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:25.863+0000 7f6f81af0700 1 -- 172.21.15.52:0/3758833266 learned_addr learned my addr 172.21.15.52:0/3758833266 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:09:25.884 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:25.863+0000 7f6f81af0700 1 -- 172.21.15.52:0/3758833266 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f7c101380 con 0x7f6f7c0703f0 2024-02-19T22:09:25.884 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:25.863+0000 7f6f81af0700 1 --2- 172.21.15.52:0/3758833266 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6f7c0703f0 0x7f6f7c100e90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f6f7800b5f0 tx=0x7f6f78006ee0).ready entity=mon.0 client_cookie=a7a5404205a9123b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:25.884 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:25.863+0000 7f6f80aee700 1 -- 172.21.15.52:0/3758833266 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f7800d600 con 0x7f6f7c0703f0 2024-02-19T22:09:25.884 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:25.863+0000 7f6f80aee700 1 -- 172.21.15.52:0/3758833266 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f6f7800dbf0 con 0x7f6f7c0703f0 2024-02-19T22:09:25.884 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:25.863+0000 7f6f80aee700 1 -- 172.21.15.52:0/3758833266 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f78011ce0 con 0x7f6f7c0703f0 2024-02-19T22:09:25.884 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:25.864+0000 7f6f83d54700 1 -- 172.21.15.52:0/3758833266 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6f7c0703f0 msgr2=0x7f6f7c100e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:25.885 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:25.864+0000 7f6f83d54700 1 --2- 172.21.15.52:0/3758833266 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6f7c0703f0 0x7f6f7c100e90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f6f7800b5f0 tx=0x7f6f78006ee0).stop 2024-02-19T22:09:25.885 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:25.864+0000 7f6f83d54700 1 -- 172.21.15.52:0/3758833266 shutdown_connections 2024-02-19T22:09:25.885 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:25.864+0000 7f6f83d54700 1 --2- 172.21.15.52:0/3758833266 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6f7c0703f0 0x7f6f7c100e90 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:25.885 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:25.864+0000 7f6f83d54700 1 -- 172.21.15.52:0/3758833266 >> 172.21.15.52:0/3758833266 conn(0x7f6f7c0f6530 msgr2=0x7f6f7c0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:25.885 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:25.864+0000 7f6f83d54700 1 -- 172.21.15.52:0/3758833266 shutdown_connections 2024-02-19T22:09:25.885 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:25.864+0000 7f6f83d54700 1 -- 172.21.15.52:0/3758833266 wait complete. 2024-02-19T22:09:25.885 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:25.865+0000 7f6f83d54700 1 Processor -- start 2024-02-19T22:09:25.885 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:25.865+0000 7f6f83d54700 1 -- start start 2024-02-19T22:09:25.885 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:25.865+0000 7f6f83d54700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6f7c0703f0 0x7f6f7c10cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:25.885 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:25.865+0000 7f6f83d54700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7c10d290 con 0x7f6f7c0703f0 2024-02-19T22:09:25.885 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:25.865+0000 7f6f81af0700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6f7c0703f0 0x7f6f7c10cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:25.885 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:25.865+0000 7f6f81af0700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6f7c0703f0 0x7f6f7c10cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:52988/0 (socket says 172.21.15.52:52988) 2024-02-19T22:09:25.885 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:25.865+0000 7f6f81af0700 1 -- 172.21.15.52:0/4173964937 learned_addr learned my addr 172.21.15.52:0/4173964937 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:09:25.886 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:25.865+0000 7f6f81af0700 1 -- 172.21.15.52:0/4173964937 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f7800b200 con 0x7f6f7c0703f0 2024-02-19T22:09:25.886 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:25.865+0000 7f6f81af0700 1 --2- 172.21.15.52:0/4173964937 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6f7c0703f0 0x7f6f7c10cda0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f6f78011ed0 tx=0x7f6f78006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:25.886 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:25.865+0000 7f6f6e7fc700 1 -- 172.21.15.52:0/4173964937 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f7801b570 con 0x7f6f7c0703f0 2024-02-19T22:09:25.886 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:25.865+0000 7f6f6e7fc700 1 -- 172.21.15.52:0/4173964937 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f6f7801bb60 con 0x7f6f7c0703f0 2024-02-19T22:09:25.886 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:25.865+0000 7f6f83d54700 1 -- 172.21.15.52:0/4173964937 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f7c10d490 con 0x7f6f7c0703f0 2024-02-19T22:09:25.886 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:25.865+0000 7f6f6e7fc700 1 -- 172.21.15.52:0/4173964937 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f7801aad0 con 0x7f6f7c0703f0 2024-02-19T22:09:25.886 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:25.865+0000 7f6f83d54700 1 -- 172.21.15.52:0/4173964937 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f7c10e0d0 con 0x7f6f7c0703f0 2024-02-19T22:09:25.886 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:25.866+0000 7f6f6e7fc700 1 -- 172.21.15.52:0/4173964937 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f6f7800f070 con 0x7f6f7c0703f0 2024-02-19T22:09:25.886 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:25.866+0000 7f6f83d54700 1 -- 172.21.15.52:0/4173964937 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f7c10d920 con 0x7f6f7c0703f0 2024-02-19T22:09:25.886 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:25.866+0000 7f6f6e7fc700 1 --2- 172.21.15.52:0/4173964937 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f6f6402ff70 0x7f6f64032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:25.886 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:25.866+0000 7f6f6e7fc700 1 -- 172.21.15.52:0/4173964937 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f6f78041540 con 0x7f6f7c0703f0 2024-02-19T22:09:25.886 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:25.866+0000 7f6f812ef700 1 --2- 172.21.15.52:0/4173964937 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f6f6402ff70 0x7f6f64032430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:25.886 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:25.867+0000 7f6f812ef700 1 --2- 172.21.15.52:0/4173964937 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f6f6402ff70 0x7f6f64032430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f6f700096a0 tx=0x7f6f70006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:25.886 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:25.871+0000 7f6f6e7fc700 1 -- 172.21.15.52:0/4173964937 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6f78025460 con 0x7f6f7c0703f0 2024-02-19T22:09:26.058 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:26.058+0000 7f6f83d54700 1 -- 172.21.15.52:0/4173964937 --> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f6f7c10e310 con 0x7f6f6402ff70 2024-02-19T22:09:26.064 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:26.061+0000 7f6f6e7fc700 1 -- 172.21.15.52:0/4173964937 <== mgr.14142 v2:172.21.15.52:6800/2071220111 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f6f7c10e310 con 0x7f6f6402ff70 2024-02-19T22:09:26.064 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-02-19T22:09:26.064 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:26.063+0000 7f6f83d54700 1 -- 172.21.15.52:0/4173964937 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f6f6402ff70 msgr2=0x7f6f64032430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:26.064 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:26.063+0000 7f6f83d54700 1 --2- 172.21.15.52:0/4173964937 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f6f6402ff70 0x7f6f64032430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f6f700096a0 tx=0x7f6f70006b40).stop 2024-02-19T22:09:26.065 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:26.063+0000 7f6f83d54700 1 -- 172.21.15.52:0/4173964937 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6f7c0703f0 msgr2=0x7f6f7c10cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:26.065 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:26.063+0000 7f6f83d54700 1 --2- 172.21.15.52:0/4173964937 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6f7c0703f0 0x7f6f7c10cda0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f6f78011ed0 tx=0x7f6f78006ee0).stop 2024-02-19T22:09:26.065 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:26.063+0000 7f6f83d54700 1 -- 172.21.15.52:0/4173964937 shutdown_connections 2024-02-19T22:09:26.065 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:26.063+0000 7f6f83d54700 1 --2- 172.21.15.52:0/4173964937 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f6f6402ff70 0x7f6f64032430 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:26.065 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:26.063+0000 7f6f83d54700 1 --2- 172.21.15.52:0/4173964937 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6f7c0703f0 0x7f6f7c10cda0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:26.065 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:26.063+0000 7f6f83d54700 1 -- 172.21.15.52:0/4173964937 >> 172.21.15.52:0/4173964937 conn(0x7f6f7c0f6530 msgr2=0x7f6f7c0f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:26.065 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:26.063+0000 7f6f83d54700 1 -- 172.21.15.52:0/4173964937 shutdown_connections 2024-02-19T22:09:26.065 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:26.063+0000 7f6f83d54700 1 -- 172.21.15.52:0/4173964937 wait complete. 2024-02-19T22:09:26.087 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:25 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:24.690551+0000 mgr.smithi052.ghqrtk (mgr.14142) 5 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:09:26.087 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:25 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cephadm 2024-02-19T22:09:24.691184+0000 mgr.smithi052.ghqrtk (mgr.14142) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2024-02-19T22:09:26.087 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:25 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:24.693295+0000 mon.smithi052 (mon.0) 80 : audit [INF] from='mgr.14142 172.21.15.52:0/284471578' entity='mgr.smithi052.ghqrtk' 2024-02-19T22:09:27.342 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:27 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:26.059121+0000 mgr.smithi052.ghqrtk (mgr.14142) 7 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:09:27.342 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:27 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cephadm 2024-02-19T22:09:26.059631+0000 mgr.smithi052.ghqrtk (mgr.14142) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2024-02-19T22:09:27.342 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:27 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:26.061805+0000 mon.smithi052 (mon.0) 81 : audit [INF] from='mgr.14142 172.21.15.52:0/284471578' entity='mgr.smithi052.ghqrtk' 2024-02-19T22:09:31.024 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi052 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphvym4zi3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp6ou7w4ul:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2024-02-19T22:09:34.774 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:33.915+0000 7f2da92ef700 1 Processor -- start 2024-02-19T22:09:34.774 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:33.915+0000 7f2da92ef700 1 -- start start 2024-02-19T22:09:34.774 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:33.915+0000 7f2da92ef700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2da40703b0 0x7f2da4100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:34.774 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:33.915+0000 7f2da92ef700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2da40fc8c0 con 0x7f2da40703b0 2024-02-19T22:09:34.775 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:33.916+0000 7f2da2ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2da40703b0 0x7f2da4100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:34.775 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:33.916+0000 7f2da2ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2da40703b0 0x7f2da4100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:52532/0 (socket says 172.21.15.52:52532) 2024-02-19T22:09:34.775 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:33.916+0000 7f2da2ffd700 1 -- 172.21.15.52:0/4216429972 learned_addr learned my addr 172.21.15.52:0/4216429972 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:09:34.775 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:33.916+0000 7f2da2ffd700 1 -- 172.21.15.52:0/4216429972 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2da41013c0 con 0x7f2da40703b0 2024-02-19T22:09:34.780 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:33.916+0000 7f2da2ffd700 1 --2- 172.21.15.52:0/4216429972 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2da40703b0 0x7f2da4100ed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f2d9400b5f0 tx=0x7f2d94006ee0).ready entity=mon.0 client_cookie=9b94e22c28c8917f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:34.780 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:33.916+0000 7f2da1ffb700 1 -- 172.21.15.52:0/4216429972 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d9400d600 con 0x7f2da40703b0 2024-02-19T22:09:34.780 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:33.916+0000 7f2da1ffb700 1 -- 172.21.15.52:0/4216429972 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f2d9400dbf0 con 0x7f2da40703b0 2024-02-19T22:09:34.780 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:33.916+0000 7f2da1ffb700 1 -- 172.21.15.52:0/4216429972 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d94011cb0 con 0x7f2da40703b0 2024-02-19T22:09:34.780 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:33.917+0000 7f2da92ef700 1 -- 172.21.15.52:0/4216429972 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2da40703b0 msgr2=0x7f2da4100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:34.780 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:33.917+0000 7f2da92ef700 1 --2- 172.21.15.52:0/4216429972 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2da40703b0 0x7f2da4100ed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f2d9400b5f0 tx=0x7f2d94006ee0).stop 2024-02-19T22:09:34.780 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:33.917+0000 7f2da92ef700 1 -- 172.21.15.52:0/4216429972 shutdown_connections 2024-02-19T22:09:34.780 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:33.917+0000 7f2da92ef700 1 --2- 172.21.15.52:0/4216429972 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2da40703b0 0x7f2da4100ed0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:34.780 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:33.917+0000 7f2da92ef700 1 -- 172.21.15.52:0/4216429972 >> 172.21.15.52:0/4216429972 conn(0x7f2da40f6530 msgr2=0x7f2da40f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:34.780 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:33.917+0000 7f2da92ef700 1 -- 172.21.15.52:0/4216429972 shutdown_connections 2024-02-19T22:09:34.781 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:33.917+0000 7f2da92ef700 1 -- 172.21.15.52:0/4216429972 wait complete. 2024-02-19T22:09:34.781 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:33.918+0000 7f2da92ef700 1 Processor -- start 2024-02-19T22:09:34.781 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:33.918+0000 7f2da92ef700 1 -- start start 2024-02-19T22:09:34.781 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:33.918+0000 7f2da92ef700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2da40703b0 0x7f2da410cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:34.781 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:33.918+0000 7f2da92ef700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2da410d290 con 0x7f2da40703b0 2024-02-19T22:09:34.781 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:33.918+0000 7f2da2ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2da40703b0 0x7f2da410cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:34.781 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:33.918+0000 7f2da2ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2da40703b0 0x7f2da410cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:52538/0 (socket says 172.21.15.52:52538) 2024-02-19T22:09:34.781 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:33.918+0000 7f2da2ffd700 1 -- 172.21.15.52:0/2516367872 learned_addr learned my addr 172.21.15.52:0/2516367872 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:09:34.781 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:33.918+0000 7f2da2ffd700 1 -- 172.21.15.52:0/2516367872 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d9400b200 con 0x7f2da40703b0 2024-02-19T22:09:34.781 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:33.918+0000 7f2da2ffd700 1 --2- 172.21.15.52:0/2516367872 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2da40703b0 0x7f2da410cda0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f2d94006800 tx=0x7f2d94006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:34.781 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:33.918+0000 7f2d8b7fe700 1 -- 172.21.15.52:0/2516367872 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d9401b570 con 0x7f2da40703b0 2024-02-19T22:09:34.781 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:33.918+0000 7f2d8b7fe700 1 -- 172.21.15.52:0/2516367872 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f2d9401bb60 con 0x7f2da40703b0 2024-02-19T22:09:34.781 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:33.918+0000 7f2d8b7fe700 1 -- 172.21.15.52:0/2516367872 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d9401a9f0 con 0x7f2da40703b0 2024-02-19T22:09:34.781 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:33.919+0000 7f2da92ef700 1 -- 172.21.15.52:0/2516367872 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2da410d490 con 0x7f2da40703b0 2024-02-19T22:09:34.781 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:33.919+0000 7f2da92ef700 1 -- 172.21.15.52:0/2516367872 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2da410e0d0 con 0x7f2da40703b0 2024-02-19T22:09:34.782 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:33.919+0000 7f2d8b7fe700 1 -- 172.21.15.52:0/2516367872 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f2d9400f070 con 0x7f2da40703b0 2024-02-19T22:09:34.782 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:33.920+0000 7f2da92ef700 1 -- 172.21.15.52:0/2516367872 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2da410d920 con 0x7f2da40703b0 2024-02-19T22:09:34.782 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:33.920+0000 7f2d8b7fe700 1 --2- 172.21.15.52:0/2516367872 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f2d9002ff20 0x7f2d900323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:34.782 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:33.920+0000 7f2d8b7fe700 1 -- 172.21.15.52:0/2516367872 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f2d940415d0 con 0x7f2da40703b0 2024-02-19T22:09:34.782 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:33.920+0000 7f2da27fc700 1 --2- 172.21.15.52:0/2516367872 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f2d9002ff20 0x7f2d900323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:34.782 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:33.920+0000 7f2da27fc700 1 --2- 172.21.15.52:0/2516367872 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f2d9002ff20 0x7f2d900323e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f2d980096a0 tx=0x7f2d98006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:34.782 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:33.924+0000 7f2d8b7fe700 1 -- 172.21.15.52:0/2516367872 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2d94010360 con 0x7f2da40703b0 2024-02-19T22:09:34.782 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:34.085+0000 7f2da92ef700 1 -- 172.21.15.52:0/2516367872 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f2da410ef00 con 0x7f2da40703b0 2024-02-19T22:09:34.782 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:34.087+0000 7f2d8b7fe700 1 -- 172.21.15.52:0/2516367872 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v8)=0 v8) v1 ==== 142+0+0 (secure 0 0 0) 0x7f2d940160d0 con 0x7f2da40703b0 2024-02-19T22:09:34.782 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:34.089+0000 7f2da92ef700 1 -- 172.21.15.52:0/2516367872 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f2d9002ff20 msgr2=0x7f2d900323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:34.782 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:34.089+0000 7f2da92ef700 1 --2- 172.21.15.52:0/2516367872 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f2d9002ff20 0x7f2d900323e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f2d980096a0 tx=0x7f2d98006b40).stop 2024-02-19T22:09:34.782 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:34.089+0000 7f2da92ef700 1 -- 172.21.15.52:0/2516367872 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2da40703b0 msgr2=0x7f2da410cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:34.782 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:34.089+0000 7f2da92ef700 1 --2- 172.21.15.52:0/2516367872 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2da40703b0 0x7f2da410cda0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f2d94006800 tx=0x7f2d94006ee0).stop 2024-02-19T22:09:34.782 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:34.089+0000 7f2da92ef700 1 -- 172.21.15.52:0/2516367872 shutdown_connections 2024-02-19T22:09:34.783 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:34.089+0000 7f2da92ef700 1 --2- 172.21.15.52:0/2516367872 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f2d9002ff20 0x7f2d900323e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:34.783 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:34.089+0000 7f2da92ef700 1 --2- 172.21.15.52:0/2516367872 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2da40703b0 0x7f2da410cda0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:34.783 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:34.089+0000 7f2da92ef700 1 -- 172.21.15.52:0/2516367872 >> 172.21.15.52:0/2516367872 conn(0x7f2da40f6530 msgr2=0x7f2da40f7f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:34.783 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:34.089+0000 7f2da92ef700 1 -- 172.21.15.52:0/2516367872 shutdown_connections 2024-02-19T22:09:34.783 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:34.089+0000 7f2da92ef700 1 -- 172.21.15.52:0/2516367872 wait complete. 2024-02-19T22:09:35.342 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:35 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:34.087934+0000 mon.smithi052 (mon.0) 82 : audit [INF] from='client.? 172.21.15.52:0/2516367872' entity='client.admin' 2024-02-19T22:09:35.342 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:35 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:34.667495+0000 mon.smithi052 (mon.0) 83 : audit [INF] from='mgr.14142 172.21.15.52:0/284471578' entity='mgr.smithi052.ghqrtk' 2024-02-19T22:09:35.342 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:35 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:35.003728+0000 mon.smithi052 (mon.0) 84 : audit [INF] from='mgr.14142 172.21.15.52:0/284471578' entity='mgr.smithi052.ghqrtk' 2024-02-19T22:09:35.396 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi052 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphvym4zi3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp6ou7w4ul:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2024-02-19T22:09:36.202 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.175+0000 7f5c43273700 1 Processor -- start 2024-02-19T22:09:36.202 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.176+0000 7f5c43273700 1 -- start start 2024-02-19T22:09:36.202 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.176+0000 7f5c43273700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5c3c070430 0x7f5c3c0fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:36.202 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.176+0000 7f5c43273700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3c0fef40 con 0x7f5c3c070430 2024-02-19T22:09:36.203 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.176+0000 7f5c4100f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5c3c070430 0x7f5c3c0fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:36.203 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.176+0000 7f5c4100f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5c3c070430 0x7f5c3c0fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:52554/0 (socket says 172.21.15.52:52554) 2024-02-19T22:09:36.203 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.176+0000 7f5c4100f700 1 -- 172.21.15.52:0/2539457785 learned_addr learned my addr 172.21.15.52:0/2539457785 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:09:36.203 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.176+0000 7f5c4100f700 1 -- 172.21.15.52:0/2539457785 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c3c0ff080 con 0x7f5c3c070430 2024-02-19T22:09:36.203 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.176+0000 7f5c4100f700 1 --2- 172.21.15.52:0/2539457785 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5c3c070430 0x7f5c3c0fea50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f5c2c00b5f0 tx=0x7f5c2c006ee0).ready entity=mon.0 client_cookie=19d4f779723311c5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:36.203 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.176+0000 7f5c2bfff700 1 -- 172.21.15.52:0/2539457785 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c2c00d600 con 0x7f5c3c070430 2024-02-19T22:09:36.203 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.176+0000 7f5c2bfff700 1 -- 172.21.15.52:0/2539457785 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5c2c00dc30 con 0x7f5c3c070430 2024-02-19T22:09:36.203 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.176+0000 7f5c2bfff700 1 -- 172.21.15.52:0/2539457785 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c2c011d30 con 0x7f5c3c070430 2024-02-19T22:09:36.203 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.177+0000 7f5c43273700 1 -- 172.21.15.52:0/2539457785 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5c3c070430 msgr2=0x7f5c3c0fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:36.203 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.177+0000 7f5c43273700 1 --2- 172.21.15.52:0/2539457785 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5c3c070430 0x7f5c3c0fea50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f5c2c00b5f0 tx=0x7f5c2c006ee0).stop 2024-02-19T22:09:36.203 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.177+0000 7f5c43273700 1 -- 172.21.15.52:0/2539457785 shutdown_connections 2024-02-19T22:09:36.203 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.177+0000 7f5c43273700 1 --2- 172.21.15.52:0/2539457785 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5c3c070430 0x7f5c3c0fea50 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:36.203 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.177+0000 7f5c43273700 1 -- 172.21.15.52:0/2539457785 >> 172.21.15.52:0/2539457785 conn(0x7f5c3c0f6530 msgr2=0x7f5c3c0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:36.203 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.177+0000 7f5c43273700 1 -- 172.21.15.52:0/2539457785 shutdown_connections 2024-02-19T22:09:36.203 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.177+0000 7f5c43273700 1 -- 172.21.15.52:0/2539457785 wait complete. 2024-02-19T22:09:36.203 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.178+0000 7f5c43273700 1 Processor -- start 2024-02-19T22:09:36.204 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.178+0000 7f5c43273700 1 -- start start 2024-02-19T22:09:36.204 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.178+0000 7f5c43273700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5c3c070430 0x7f5c3c10ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:36.204 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.178+0000 7f5c43273700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3c10b060 con 0x7f5c3c070430 2024-02-19T22:09:36.204 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.178+0000 7f5c4100f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5c3c070430 0x7f5c3c10ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:36.204 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.178+0000 7f5c4100f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5c3c070430 0x7f5c3c10ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:52558/0 (socket says 172.21.15.52:52558) 2024-02-19T22:09:36.204 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.178+0000 7f5c4100f700 1 -- 172.21.15.52:0/2747828154 learned_addr learned my addr 172.21.15.52:0/2747828154 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:09:36.204 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.178+0000 7f5c4100f700 1 -- 172.21.15.52:0/2747828154 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c2c00b200 con 0x7f5c3c070430 2024-02-19T22:09:36.204 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.178+0000 7f5c4100f700 1 --2- 172.21.15.52:0/2747828154 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5c3c070430 0x7f5c3c10ab70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f5c2c00e040 tx=0x7f5c2c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:36.204 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.178+0000 7f5c29ffb700 1 -- 172.21.15.52:0/2747828154 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c2c00da60 con 0x7f5c3c070430 2024-02-19T22:09:36.204 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.178+0000 7f5c29ffb700 1 -- 172.21.15.52:0/2747828154 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5c2c01b8e0 con 0x7f5c3c070430 2024-02-19T22:09:36.204 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.178+0000 7f5c29ffb700 1 -- 172.21.15.52:0/2747828154 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c2c01a900 con 0x7f5c3c070430 2024-02-19T22:09:36.204 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.178+0000 7f5c43273700 1 -- 172.21.15.52:0/2747828154 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c3c10b260 con 0x7f5c3c070430 2024-02-19T22:09:36.204 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.179+0000 7f5c43273700 1 -- 172.21.15.52:0/2747828154 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c3c10b6f0 con 0x7f5c3c070430 2024-02-19T22:09:36.204 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.179+0000 7f5c29ffb700 1 -- 172.21.15.52:0/2747828154 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f5c2c00f070 con 0x7f5c3c070430 2024-02-19T22:09:36.204 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.179+0000 7f5c29ffb700 1 --2- 172.21.15.52:0/2747828154 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f5c2402ff70 0x7f5c24032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:36.205 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.179+0000 7f5c29ffb700 1 -- 172.21.15.52:0/2747828154 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f5c2c0413f0 con 0x7f5c3c070430 2024-02-19T22:09:36.205 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.180+0000 7f5c4080e700 1 --2- 172.21.15.52:0/2747828154 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f5c2402ff70 0x7f5c24032430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:36.205 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.180+0000 7f5c43273700 1 -- 172.21.15.52:0/2747828154 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c3c045ad0 con 0x7f5c3c070430 2024-02-19T22:09:36.205 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.180+0000 7f5c4080e700 1 --2- 172.21.15.52:0/2747828154 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f5c2402ff70 0x7f5c24032430 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f5c380096a0 tx=0x7f5c38006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:36.205 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.184+0000 7f5c29ffb700 1 -- 172.21.15.52:0/2747828154 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5c2c016070 con 0x7f5c3c070430 2024-02-19T22:09:36.345 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.344+0000 7f5c43273700 1 -- 172.21.15.52:0/2747828154 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f5c3c104bd0 con 0x7f5c3c070430 2024-02-19T22:09:36.348 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.347+0000 7f5c29ffb700 1 -- 172.21.15.52:0/2747828154 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v10)=0 v10) v1 ==== 130+0+0 (secure 0 0 0) 0x7f5c2c0224f0 con 0x7f5c3c070430 2024-02-19T22:09:36.351 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.348+0000 7f5c43273700 1 -- 172.21.15.52:0/2747828154 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f5c2402ff70 msgr2=0x7f5c24032430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:36.351 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.348+0000 7f5c43273700 1 --2- 172.21.15.52:0/2747828154 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f5c2402ff70 0x7f5c24032430 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f5c380096a0 tx=0x7f5c38006b40).stop 2024-02-19T22:09:36.351 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.348+0000 7f5c43273700 1 -- 172.21.15.52:0/2747828154 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5c3c070430 msgr2=0x7f5c3c10ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:36.351 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.348+0000 7f5c43273700 1 --2- 172.21.15.52:0/2747828154 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5c3c070430 0x7f5c3c10ab70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f5c2c00e040 tx=0x7f5c2c006ee0).stop 2024-02-19T22:09:36.351 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.349+0000 7f5c43273700 1 -- 172.21.15.52:0/2747828154 shutdown_connections 2024-02-19T22:09:36.351 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.349+0000 7f5c43273700 1 --2- 172.21.15.52:0/2747828154 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f5c2402ff70 0x7f5c24032430 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:36.351 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.349+0000 7f5c43273700 1 --2- 172.21.15.52:0/2747828154 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5c3c070430 0x7f5c3c10ab70 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:36.351 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.349+0000 7f5c43273700 1 -- 172.21.15.52:0/2747828154 >> 172.21.15.52:0/2747828154 conn(0x7f5c3c0f6530 msgr2=0x7f5c3c0f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:36.351 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.349+0000 7f5c43273700 1 -- 172.21.15.52:0/2747828154 shutdown_connections 2024-02-19T22:09:36.351 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:36.349+0000 7f5c43273700 1 -- 172.21.15.52:0/2747828154 wait complete. 2024-02-19T22:09:37.265 INFO:teuthology.orchestra.run.smithi052.stderr:Enabling the dashboard module... 2024-02-19T22:09:37.265 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi052 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphvym4zi3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp6ou7w4ul:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2024-02-19T22:09:37.522 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:37 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:36.348243+0000 mon.smithi052 (mon.0) 85 : audit [INF] from='client.? 172.21.15.52:0/2747828154' entity='client.admin' 2024-02-19T22:09:37.522 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:37 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:36.831218+0000 mon.smithi052 (mon.0) 86 : audit [INF] from='mgr.14142 172.21.15.52:0/284471578' entity='mgr.smithi052.ghqrtk' 2024-02-19T22:09:38.088 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.009+0000 7f50eb70a700 1 Processor -- start 2024-02-19T22:09:38.089 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.009+0000 7f50eb70a700 1 -- start start 2024-02-19T22:09:38.089 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.009+0000 7f50eb70a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50e40703d0 0x7f50e4100eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:38.089 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.009+0000 7f50eb70a700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e40fc8a0 con 0x7f50e40703d0 2024-02-19T22:09:38.089 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.009+0000 7f50e94a6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50e40703d0 0x7f50e4100eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:38.089 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.009+0000 7f50e94a6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50e40703d0 0x7f50e4100eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:52572/0 (socket says 172.21.15.52:52572) 2024-02-19T22:09:38.089 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.009+0000 7f50e94a6700 1 -- 172.21.15.52:0/1672405259 learned_addr learned my addr 172.21.15.52:0/1672405259 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:09:38.089 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.009+0000 7f50e94a6700 1 -- 172.21.15.52:0/1672405259 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e41013a0 con 0x7f50e40703d0 2024-02-19T22:09:38.089 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.010+0000 7f50e94a6700 1 --2- 172.21.15.52:0/1672405259 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50e40703d0 0x7f50e4100eb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f50d400b5f0 tx=0x7f50d4006ee0).ready entity=mon.0 client_cookie=5b6287e956092a55 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:38.089 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.010+0000 7f50d3fff700 1 -- 172.21.15.52:0/1672405259 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50d400d600 con 0x7f50e40703d0 2024-02-19T22:09:38.089 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.010+0000 7f50d3fff700 1 -- 172.21.15.52:0/1672405259 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f50d400dc30 con 0x7f50e40703d0 2024-02-19T22:09:38.089 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.010+0000 7f50d3fff700 1 -- 172.21.15.52:0/1672405259 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50d4011d30 con 0x7f50e40703d0 2024-02-19T22:09:38.090 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.010+0000 7f50eb70a700 1 -- 172.21.15.52:0/1672405259 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50e40703d0 msgr2=0x7f50e4100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:38.090 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.010+0000 7f50eb70a700 1 --2- 172.21.15.52:0/1672405259 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50e40703d0 0x7f50e4100eb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f50d400b5f0 tx=0x7f50d4006ee0).stop 2024-02-19T22:09:38.090 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.010+0000 7f50eb70a700 1 -- 172.21.15.52:0/1672405259 shutdown_connections 2024-02-19T22:09:38.090 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.010+0000 7f50eb70a700 1 --2- 172.21.15.52:0/1672405259 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50e40703d0 0x7f50e4100eb0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:38.090 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.010+0000 7f50eb70a700 1 -- 172.21.15.52:0/1672405259 >> 172.21.15.52:0/1672405259 conn(0x7f50e40f6530 msgr2=0x7f50e40f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:38.090 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.010+0000 7f50eb70a700 1 -- 172.21.15.52:0/1672405259 shutdown_connections 2024-02-19T22:09:38.090 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.010+0000 7f50eb70a700 1 -- 172.21.15.52:0/1672405259 wait complete. 2024-02-19T22:09:38.090 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.011+0000 7f50eb70a700 1 Processor -- start 2024-02-19T22:09:38.090 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.011+0000 7f50eb70a700 1 -- start start 2024-02-19T22:09:38.090 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.011+0000 7f50eb70a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50e40703d0 0x7f50e410cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:38.090 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.011+0000 7f50eb70a700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e410d3b0 con 0x7f50e40703d0 2024-02-19T22:09:38.090 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.011+0000 7f50e94a6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50e40703d0 0x7f50e410cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:38.090 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.011+0000 7f50e94a6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50e40703d0 0x7f50e410cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:52584/0 (socket says 172.21.15.52:52584) 2024-02-19T22:09:38.090 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.011+0000 7f50e94a6700 1 -- 172.21.15.52:0/1300255997 learned_addr learned my addr 172.21.15.52:0/1300255997 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:09:38.090 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.011+0000 7f50e94a6700 1 -- 172.21.15.52:0/1300255997 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50d400b200 con 0x7f50e40703d0 2024-02-19T22:09:38.091 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.011+0000 7f50e94a6700 1 --2- 172.21.15.52:0/1300255997 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50e40703d0 0x7f50e410cec0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f50d4006c70 tx=0x7f50d4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:38.091 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.012+0000 7f50d1ffb700 1 -- 172.21.15.52:0/1300255997 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50d400d850 con 0x7f50e40703d0 2024-02-19T22:09:38.091 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.012+0000 7f50d1ffb700 1 -- 172.21.15.52:0/1300255997 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f50d400d9b0 con 0x7f50e40703d0 2024-02-19T22:09:38.091 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.012+0000 7f50eb70a700 1 -- 172.21.15.52:0/1300255997 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50e410d5b0 con 0x7f50e40703d0 2024-02-19T22:09:38.091 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.012+0000 7f50d1ffb700 1 -- 172.21.15.52:0/1300255997 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50d401a750 con 0x7f50e40703d0 2024-02-19T22:09:38.091 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.012+0000 7f50eb70a700 1 -- 172.21.15.52:0/1300255997 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50e410e160 con 0x7f50e40703d0 2024-02-19T22:09:38.091 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.013+0000 7f50d1ffb700 1 -- 172.21.15.52:0/1300255997 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f50d400f070 con 0x7f50e40703d0 2024-02-19T22:09:38.091 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.013+0000 7f50d1ffb700 1 --2- 172.21.15.52:0/1300255997 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f50cc02ff20 0x7f50cc0323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:38.091 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.013+0000 7f50eb70a700 1 -- 172.21.15.52:0/1300255997 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50e4059100 con 0x7f50e40703d0 2024-02-19T22:09:38.091 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.013+0000 7f50d1ffb700 1 -- 172.21.15.52:0/1300255997 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f50d4040e20 con 0x7f50e40703d0 2024-02-19T22:09:38.091 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.013+0000 7f50e8ca5700 1 --2- 172.21.15.52:0/1300255997 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f50cc02ff20 0x7f50cc0323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:38.091 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.013+0000 7f50e8ca5700 1 --2- 172.21.15.52:0/1300255997 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f50cc02ff20 0x7f50cc0323e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f50e00096a0 tx=0x7f50e0006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:38.091 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.017+0000 7f50d1ffb700 1 -- 172.21.15.52:0/1300255997 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f50d4016070 con 0x7f50e40703d0 2024-02-19T22:09:38.224 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.223+0000 7f50eb70a700 1 -- 172.21.15.52:0/1300255997 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f50e41068d0 con 0x7f50e40703d0 2024-02-19T22:09:38.350 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.349+0000 7f50d1ffb700 1 -- 172.21.15.52:0/1300255997 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f50d4010dd0 con 0x7f50e40703d0 2024-02-19T22:09:38.351 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.350+0000 7f50d1ffb700 1 -- 172.21.15.52:0/1300255997 <== mon.0 v2:172.21.15.52:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v13) v1 ==== 88+0+0 (secure 0 0 0) 0x7f50d4019590 con 0x7f50e40703d0 2024-02-19T22:09:38.354 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.352+0000 7f50eb70a700 1 -- 172.21.15.52:0/1300255997 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f50cc02ff20 msgr2=0x7f50cc0323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:38.354 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.352+0000 7f50eb70a700 1 --2- 172.21.15.52:0/1300255997 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f50cc02ff20 0x7f50cc0323e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f50e00096a0 tx=0x7f50e0006b40).stop 2024-02-19T22:09:38.355 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.352+0000 7f50eb70a700 1 -- 172.21.15.52:0/1300255997 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50e40703d0 msgr2=0x7f50e410cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:38.355 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.352+0000 7f50eb70a700 1 --2- 172.21.15.52:0/1300255997 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50e40703d0 0x7f50e410cec0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f50d4006c70 tx=0x7f50d4006ee0).stop 2024-02-19T22:09:38.355 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.352+0000 7f50eb70a700 1 -- 172.21.15.52:0/1300255997 shutdown_connections 2024-02-19T22:09:38.355 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.352+0000 7f50eb70a700 1 --2- 172.21.15.52:0/1300255997 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f50cc02ff20 0x7f50cc0323e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:38.355 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.352+0000 7f50eb70a700 1 --2- 172.21.15.52:0/1300255997 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50e40703d0 0x7f50e410cec0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:38.355 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.352+0000 7f50eb70a700 1 -- 172.21.15.52:0/1300255997 >> 172.21.15.52:0/1300255997 conn(0x7f50e40f6530 msgr2=0x7f50e40f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:38.355 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.352+0000 7f50eb70a700 1 -- 172.21.15.52:0/1300255997 shutdown_connections 2024-02-19T22:09:38.355 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:38.352+0000 7f50eb70a700 1 -- 172.21.15.52:0/1300255997 wait complete. 2024-02-19T22:09:38.841 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:38 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:38.224796+0000 mon.smithi052 (mon.0) 87 : audit [INF] from='client.? 172.21.15.52:0/1300255997' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-19T22:09:39.194 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi052 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphvym4zi3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp6ou7w4ul:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-02-19T22:09:39.459 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:38.351436+0000 mon.smithi052 (mon.0) 88 : audit [INF] from='client.? 172.21.15.52:0/1300255997' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-19T22:09:39.459 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:09:38.351521+0000 mon.smithi052 (mon.0) 89 : cluster [DBG] mgrmap e13: smithi052.ghqrtk(active, since 17s) 2024-02-19T22:09:39.968 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:39.871+0000 7f1f67261700 1 Processor -- start 2024-02-19T22:09:39.968 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:39.871+0000 7f1f67261700 1 -- start start 2024-02-19T22:09:39.968 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:39.872+0000 7f1f67261700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1f60072490 0x7f1f60070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:39.968 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:39.872+0000 7f1f67261700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f600728b0 con 0x7f1f60072490 2024-02-19T22:09:39.968 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:39.872+0000 7f1f64ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1f60072490 0x7f1f60070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:39.968 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:39.872+0000 7f1f64ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1f60072490 0x7f1f60070a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:52614/0 (socket says 172.21.15.52:52614) 2024-02-19T22:09:39.968 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:39.872+0000 7f1f64ffd700 1 -- 172.21.15.52:0/1936395497 learned_addr learned my addr 172.21.15.52:0/1936395497 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:09:39.968 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:39.872+0000 7f1f64ffd700 1 -- 172.21.15.52:0/1936395497 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f60070f20 con 0x7f1f60072490 2024-02-19T22:09:39.968 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:39.872+0000 7f1f64ffd700 1 --2- 172.21.15.52:0/1936395497 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1f60072490 0x7f1f60070a30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f1f5400b5f0 tx=0x7f1f54006ee0).ready entity=mon.0 client_cookie=603c20a816191385 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:39.968 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:39.872+0000 7f1f5f7fe700 1 -- 172.21.15.52:0/1936395497 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f5400d600 con 0x7f1f60072490 2024-02-19T22:09:39.968 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:39.872+0000 7f1f5f7fe700 1 -- 172.21.15.52:0/1936395497 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1f5400dc30 con 0x7f1f60072490 2024-02-19T22:09:39.969 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:39.872+0000 7f1f5f7fe700 1 -- 172.21.15.52:0/1936395497 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f54011d30 con 0x7f1f60072490 2024-02-19T22:09:39.969 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:39.873+0000 7f1f67261700 1 -- 172.21.15.52:0/1936395497 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1f60072490 msgr2=0x7f1f60070a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:39.969 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:39.873+0000 7f1f67261700 1 --2- 172.21.15.52:0/1936395497 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1f60072490 0x7f1f60070a30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f1f5400b5f0 tx=0x7f1f54006ee0).stop 2024-02-19T22:09:39.969 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:39.873+0000 7f1f67261700 1 -- 172.21.15.52:0/1936395497 shutdown_connections 2024-02-19T22:09:39.969 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:39.873+0000 7f1f67261700 1 --2- 172.21.15.52:0/1936395497 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1f60072490 0x7f1f60070a30 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:39.969 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:39.873+0000 7f1f67261700 1 -- 172.21.15.52:0/1936395497 >> 172.21.15.52:0/1936395497 conn(0x7f1f600f7640 msgr2=0x7f1f600f9a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:39.969 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:39.873+0000 7f1f67261700 1 -- 172.21.15.52:0/1936395497 shutdown_connections 2024-02-19T22:09:39.969 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:39.873+0000 7f1f67261700 1 -- 172.21.15.52:0/1936395497 wait complete. 2024-02-19T22:09:39.969 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:39.874+0000 7f1f67261700 1 Processor -- start 2024-02-19T22:09:39.969 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:39.874+0000 7f1f67261700 1 -- start start 2024-02-19T22:09:39.969 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:39.874+0000 7f1f67261700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1f60072490 0x7f1f6010cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:39.969 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:39.874+0000 7f1f67261700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6010d390 con 0x7f1f60072490 2024-02-19T22:09:39.969 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:39.874+0000 7f1f64ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1f60072490 0x7f1f6010cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:39.969 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:39.874+0000 7f1f64ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1f60072490 0x7f1f6010cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:52630/0 (socket says 172.21.15.52:52630) 2024-02-19T22:09:39.969 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:39.874+0000 7f1f64ffd700 1 -- 172.21.15.52:0/302538637 learned_addr learned my addr 172.21.15.52:0/302538637 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:09:39.970 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:39.874+0000 7f1f64ffd700 1 -- 172.21.15.52:0/302538637 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f5400b200 con 0x7f1f60072490 2024-02-19T22:09:39.970 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:39.874+0000 7f1f64ffd700 1 --2- 172.21.15.52:0/302538637 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1f60072490 0x7f1f6010cea0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f1f5400b1b0 tx=0x7f1f54006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:39.970 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:39.875+0000 7f1f5d7fa700 1 -- 172.21.15.52:0/302538637 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f5400d960 con 0x7f1f60072490 2024-02-19T22:09:39.970 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:39.875+0000 7f1f5d7fa700 1 -- 172.21.15.52:0/302538637 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1f5400dac0 con 0x7f1f60072490 2024-02-19T22:09:39.970 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:39.875+0000 7f1f67261700 1 -- 172.21.15.52:0/302538637 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f6010d590 con 0x7f1f60072490 2024-02-19T22:09:39.970 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:39.875+0000 7f1f5d7fa700 1 -- 172.21.15.52:0/302538637 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f5401a7c0 con 0x7f1f60072490 2024-02-19T22:09:39.970 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:39.875+0000 7f1f67261700 1 -- 172.21.15.52:0/302538637 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f6010e140 con 0x7f1f60072490 2024-02-19T22:09:39.970 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:39.875+0000 7f1f5d7fa700 1 -- 172.21.15.52:0/302538637 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f1f5400f070 con 0x7f1f60072490 2024-02-19T22:09:39.970 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:39.876+0000 7f1f5d7fa700 1 --2- 172.21.15.52:0/302538637 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f1f4402ff70 0x7f1f44032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:39.970 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:39.876+0000 7f1f5d7fa700 1 -- 172.21.15.52:0/302538637 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f1f54041330 con 0x7f1f60072490 2024-02-19T22:09:39.970 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:39.876+0000 7f1f5ffff700 1 -- 172.21.15.52:0/302538637 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f1f4402ff70 msgr2=0x7f1f44032430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/2071220111 2024-02-19T22:09:39.970 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:39.876+0000 7f1f5ffff700 1 --2- 172.21.15.52:0/302538637 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f1f4402ff70 0x7f1f44032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-19T22:09:39.970 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:39.876+0000 7f1f67261700 1 -- 172.21.15.52:0/302538637 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f60059100 con 0x7f1f60072490 2024-02-19T22:09:39.970 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:39.880+0000 7f1f5d7fa700 1 -- 172.21.15.52:0/302538637 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1f54025460 con 0x7f1f60072490 2024-02-19T22:09:40.078 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:40.077+0000 7f1f5ffff700 1 -- 172.21.15.52:0/302538637 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f1f4402ff70 msgr2=0x7f1f44032430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/2071220111 2024-02-19T22:09:40.078 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:40.077+0000 7f1f5ffff700 1 --2- 172.21.15.52:0/302538637 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f1f4402ff70 0x7f1f44032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-19T22:09:40.082 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:40.081+0000 7f1f67261700 1 -- 172.21.15.52:0/302538637 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f1f601021e0 con 0x7f1f60072490 2024-02-19T22:09:40.083 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:40.081+0000 7f1f5d7fa700 1 -- 172.21.15.52:0/302538637 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v13) v1 ==== 56+0+104 (secure 0 0 0) 0x7f1f54022310 con 0x7f1f60072490 2024-02-19T22:09:40.083 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: { 2024-02-19T22:09:40.083 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "epoch": 13, 2024-02-19T22:09:40.083 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "available": true, 2024-02-19T22:09:40.084 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "active_name": "smithi052.ghqrtk", 2024-02-19T22:09:40.084 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-19T22:09:40.084 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: } 2024-02-19T22:09:40.085 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:40.083+0000 7f1f67261700 1 -- 172.21.15.52:0/302538637 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f1f4402ff70 msgr2=0x7f1f44032430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T22:09:40.085 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:40.083+0000 7f1f67261700 1 --2- 172.21.15.52:0/302538637 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f1f4402ff70 0x7f1f44032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T22:09:40.086 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:40.083+0000 7f1f67261700 1 -- 172.21.15.52:0/302538637 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1f60072490 msgr2=0x7f1f6010cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:40.086 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:40.083+0000 7f1f67261700 1 --2- 172.21.15.52:0/302538637 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1f60072490 0x7f1f6010cea0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f1f5400b1b0 tx=0x7f1f54006ee0).stop 2024-02-19T22:09:40.086 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:40.083+0000 7f1f67261700 1 -- 172.21.15.52:0/302538637 shutdown_connections 2024-02-19T22:09:40.086 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:40.083+0000 7f1f67261700 1 --2- 172.21.15.52:0/302538637 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7f1f4402ff70 0x7f1f44032430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T22:09:40.086 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:40.083+0000 7f1f67261700 1 --2- 172.21.15.52:0/302538637 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1f60072490 0x7f1f6010cea0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:40.086 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:40.083+0000 7f1f67261700 1 -- 172.21.15.52:0/302538637 >> 172.21.15.52:0/302538637 conn(0x7f1f600f7640 msgr2=0x7f1f600f9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:40.086 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:40.083+0000 7f1f67261700 1 -- 172.21.15.52:0/302538637 shutdown_connections 2024-02-19T22:09:40.086 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:40.083+0000 7f1f67261700 1 -- 172.21.15.52:0/302538637 wait complete. 2024-02-19T22:09:40.803 INFO:teuthology.orchestra.run.smithi052.stderr:Waiting for the mgr to restart... 2024-02-19T22:09:40.803 INFO:teuthology.orchestra.run.smithi052.stderr:Waiting for mgr epoch 13... 2024-02-19T22:09:40.803 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi052 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphvym4zi3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp6ou7w4ul:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-02-19T22:09:40.805 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:40 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:40.082996+0000 mon.smithi052 (mon.0) 90 : audit [DBG] from='client.? 172.21.15.52:0/302538637' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-19T22:09:41.393 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:41.390+0000 7fd5f00b8700 1 Processor -- start 2024-02-19T22:09:41.393 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:41.390+0000 7fd5f00b8700 1 -- start start 2024-02-19T22:09:41.393 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:41.391+0000 7fd5f00b8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd5e80703b0 0x7fd5e8100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:41.393 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:41.391+0000 7fd5f00b8700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e80fc8c0 con 0x7fd5e80703b0 2024-02-19T22:09:41.394 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:41.391+0000 7fd5ede54700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd5e80703b0 0x7fd5e8100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:41.394 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:41.391+0000 7fd5ede54700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd5e80703b0 0x7fd5e8100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:52638/0 (socket says 172.21.15.52:52638) 2024-02-19T22:09:41.394 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:41.391+0000 7fd5ede54700 1 -- 172.21.15.52:0/1967326706 learned_addr learned my addr 172.21.15.52:0/1967326706 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:09:41.394 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:41.391+0000 7fd5ede54700 1 -- 172.21.15.52:0/1967326706 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5e81013c0 con 0x7fd5e80703b0 2024-02-19T22:09:41.394 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:41.391+0000 7fd5ede54700 1 --2- 172.21.15.52:0/1967326706 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd5e80703b0 0x7fd5e8100ed0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fd5e400b5f0 tx=0x7fd5e4006ee0).ready entity=mon.0 client_cookie=26979b90bc6e9427 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:41.394 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:41.391+0000 7fd5ece52700 1 -- 172.21.15.52:0/1967326706 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5e400d600 con 0x7fd5e80703b0 2024-02-19T22:09:41.394 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:41.391+0000 7fd5ece52700 1 -- 172.21.15.52:0/1967326706 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd5e400dc30 con 0x7fd5e80703b0 2024-02-19T22:09:41.394 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:41.391+0000 7fd5ece52700 1 -- 172.21.15.52:0/1967326706 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5e4011d30 con 0x7fd5e80703b0 2024-02-19T22:09:41.397 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:41.392+0000 7fd5f00b8700 1 -- 172.21.15.52:0/1967326706 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd5e80703b0 msgr2=0x7fd5e8100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:41.397 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:41.392+0000 7fd5f00b8700 1 --2- 172.21.15.52:0/1967326706 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd5e80703b0 0x7fd5e8100ed0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fd5e400b5f0 tx=0x7fd5e4006ee0).stop 2024-02-19T22:09:41.397 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:41.392+0000 7fd5f00b8700 1 -- 172.21.15.52:0/1967326706 shutdown_connections 2024-02-19T22:09:41.397 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:41.392+0000 7fd5f00b8700 1 --2- 172.21.15.52:0/1967326706 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd5e80703b0 0x7fd5e8100ed0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:41.397 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:41.393+0000 7fd5f00b8700 1 -- 172.21.15.52:0/1967326706 >> 172.21.15.52:0/1967326706 conn(0x7fd5e80f6530 msgr2=0x7fd5e80f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:41.398 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:41.393+0000 7fd5f00b8700 1 -- 172.21.15.52:0/1967326706 shutdown_connections 2024-02-19T22:09:41.398 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:41.393+0000 7fd5f00b8700 1 -- 172.21.15.52:0/1967326706 wait complete. 2024-02-19T22:09:41.398 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:41.393+0000 7fd5f00b8700 1 Processor -- start 2024-02-19T22:09:41.398 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:41.393+0000 7fd5f00b8700 1 -- start start 2024-02-19T22:09:41.398 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:41.393+0000 7fd5f00b8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd5e80703b0 0x7fd5e810cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:41.398 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:41.393+0000 7fd5f00b8700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e810d290 con 0x7fd5e80703b0 2024-02-19T22:09:41.398 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:41.394+0000 7fd5ede54700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd5e80703b0 0x7fd5e810cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:41.398 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:41.394+0000 7fd5ede54700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd5e80703b0 0x7fd5e810cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:52652/0 (socket says 172.21.15.52:52652) 2024-02-19T22:09:41.398 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:41.394+0000 7fd5ede54700 1 -- 172.21.15.52:0/463735299 learned_addr learned my addr 172.21.15.52:0/463735299 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:09:41.398 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:41.394+0000 7fd5ede54700 1 -- 172.21.15.52:0/463735299 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5e400b200 con 0x7fd5e80703b0 2024-02-19T22:09:41.398 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:41.394+0000 7fd5ede54700 1 --2- 172.21.15.52:0/463735299 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd5e80703b0 0x7fd5e810cda0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fd5e400e040 tx=0x7fd5e4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:41.398 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:41.394+0000 7fd5d67fc700 1 -- 172.21.15.52:0/463735299 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5e400da60 con 0x7fd5e80703b0 2024-02-19T22:09:41.398 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:41.394+0000 7fd5f00b8700 1 -- 172.21.15.52:0/463735299 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5e810d490 con 0x7fd5e80703b0 2024-02-19T22:09:41.399 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:41.394+0000 7fd5d67fc700 1 -- 172.21.15.52:0/463735299 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd5e401b8e0 con 0x7fd5e80703b0 2024-02-19T22:09:41.399 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:41.394+0000 7fd5d67fc700 1 -- 172.21.15.52:0/463735299 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5e401a900 con 0x7fd5e80703b0 2024-02-19T22:09:41.399 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:41.394+0000 7fd5f00b8700 1 -- 172.21.15.52:0/463735299 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5e810e0d0 con 0x7fd5e80703b0 2024-02-19T22:09:41.399 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:41.395+0000 7fd5d67fc700 1 -- 172.21.15.52:0/463735299 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fd5e400f070 con 0x7fd5e80703b0 2024-02-19T22:09:41.399 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:41.395+0000 7fd5d67fc700 1 --2- 172.21.15.52:0/463735299 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7fd5d002ff70 0x7fd5d0032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:41.399 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:41.395+0000 7fd5d67fc700 1 -- 172.21.15.52:0/463735299 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fd5e4041480 con 0x7fd5e80703b0 2024-02-19T22:09:41.399 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:41.395+0000 7fd5ed653700 1 -- 172.21.15.52:0/463735299 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7fd5d002ff70 msgr2=0x7fd5d0032430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/2071220111 2024-02-19T22:09:41.399 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:41.396+0000 7fd5ed653700 1 --2- 172.21.15.52:0/463735299 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7fd5d002ff70 0x7fd5d0032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-19T22:09:41.399 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:41.396+0000 7fd5f00b8700 1 -- 172.21.15.52:0/463735299 --> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd5e8059100 con 0x7fd5d002ff70 2024-02-19T22:09:41.598 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:41.597+0000 7fd5ed653700 1 -- 172.21.15.52:0/463735299 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7fd5d002ff70 msgr2=0x7fd5d0032430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/2071220111 2024-02-19T22:09:41.598 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:41.597+0000 7fd5ed653700 1 --2- 172.21.15.52:0/463735299 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7fd5d002ff70 0x7fd5d0032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-19T22:09:41.999 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:41.998+0000 7fd5ed653700 1 -- 172.21.15.52:0/463735299 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7fd5d002ff70 msgr2=0x7fd5d0032430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/2071220111 2024-02-19T22:09:41.999 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:41.998+0000 7fd5ed653700 1 --2- 172.21.15.52:0/463735299 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7fd5d002ff70 0x7fd5d0032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-19T22:09:42.800 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:42.799+0000 7fd5ed653700 1 -- 172.21.15.52:0/463735299 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7fd5d002ff70 msgr2=0x7fd5d0032430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/2071220111 2024-02-19T22:09:42.800 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:42.799+0000 7fd5ed653700 1 --2- 172.21.15.52:0/463735299 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7fd5d002ff70 0x7fd5d0032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-02-19T22:09:43.607 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:43.605+0000 7fd5d67fc700 1 -- 172.21.15.52:0/463735299 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mgrmap(e 14) v1 ==== 35997+0+0 (secure 0 0 0) 0x7fd5e4025980 con 0x7fd5e80703b0 2024-02-19T22:09:43.607 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:43.605+0000 7fd5d67fc700 1 -- 172.21.15.52:0/463735299 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7fd5d002ff70 msgr2=0x7fd5d0032430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T22:09:43.607 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:43.605+0000 7fd5d67fc700 1 --2- 172.21.15.52:0/463735299 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7fd5d002ff70 0x7fd5d0032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T22:09:43.745 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:43 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:09:43.554066+0000 mon.smithi052 (mon.0) 91 : cluster [INF] Active manager daemon smithi052.ghqrtk restarted 2024-02-19T22:09:43.746 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:43 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:09:43.554444+0000 mon.smithi052 (mon.0) 92 : cluster [INF] Activating manager daemon smithi052.ghqrtk 2024-02-19T22:09:43.746 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:43 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:09:43.556185+0000 mon.smithi052 (mon.0) 93 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-19T22:09:44.609 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:44.607+0000 7fd5d67fc700 1 -- 172.21.15.52:0/463735299 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7fd5e4041190 con 0x7fd5e80703b0 2024-02-19T22:09:44.610 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:44.608+0000 7fd5d67fc700 1 --2- 172.21.15.52:0/463735299 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd5d0033a50 0x7fd5d0035e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:44.610 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:44.608+0000 7fd5d67fc700 1 -- 172.21.15.52:0/463735299 --> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd5e8059100 con 0x7fd5d0033a50 2024-02-19T22:09:44.610 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:44.609+0000 7fd5ed653700 1 --2- 172.21.15.52:0/463735299 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd5d0033a50 0x7fd5d0035e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:44.610 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:44.609+0000 7fd5ed653700 1 --2- 172.21.15.52:0/463735299 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd5d0033a50 0x7fd5d0035e40 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fd5d80036d0 tx=0x7fd5d8006ca0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:44.611 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:44.610+0000 7fd5d67fc700 1 -- 172.21.15.52:0/463735299 <== mgr.14162 v2:172.21.15.52:6800/539827537 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7fd5e8059100 con 0x7fd5d0033a50 2024-02-19T22:09:44.624 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:44.621+0000 7fd5f00b8700 1 -- 172.21.15.52:0/463735299 --> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fd5e810d920 con 0x7fd5d0033a50 2024-02-19T22:09:44.624 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:44.621+0000 7fd5d67fc700 1 -- 172.21.15.52:0/463735299 <== mgr.14162 v2:172.21.15.52:6800/539827537 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fd5e810d920 con 0x7fd5d0033a50 2024-02-19T22:09:44.624 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: { 2024-02-19T22:09:44.624 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "mgrmap_epoch": 15, 2024-02-19T22:09:44.624 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: "initialized": true 2024-02-19T22:09:44.624 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: } 2024-02-19T22:09:44.624 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:44.621+0000 7fd5f00b8700 1 -- 172.21.15.52:0/463735299 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd5d0033a50 msgr2=0x7fd5d0035e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:44.624 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:44.622+0000 7fd5f00b8700 1 --2- 172.21.15.52:0/463735299 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd5d0033a50 0x7fd5d0035e40 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fd5d80036d0 tx=0x7fd5d8006ca0).stop 2024-02-19T22:09:44.624 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:44.622+0000 7fd5f00b8700 1 -- 172.21.15.52:0/463735299 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd5e80703b0 msgr2=0x7fd5e810cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:44.625 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:44.622+0000 7fd5f00b8700 1 --2- 172.21.15.52:0/463735299 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd5e80703b0 0x7fd5e810cda0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fd5e400e040 tx=0x7fd5e4006ee0).stop 2024-02-19T22:09:44.625 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:44.622+0000 7fd5f00b8700 1 -- 172.21.15.52:0/463735299 shutdown_connections 2024-02-19T22:09:44.625 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:44.622+0000 7fd5f00b8700 1 --2- 172.21.15.52:0/463735299 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd5d0033a50 0x7fd5d0035e40 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:44.625 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:44.622+0000 7fd5f00b8700 1 --2- 172.21.15.52:0/463735299 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd5e80703b0 0x7fd5e810cda0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:44.625 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:44.622+0000 7fd5f00b8700 1 --2- 172.21.15.52:0/463735299 >> [v2:172.21.15.52:6800/2071220111,v1:172.21.15.52:6801/2071220111] conn(0x7fd5d002ff70 0x7fd5d0032430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T22:09:44.625 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:44.622+0000 7fd5f00b8700 1 -- 172.21.15.52:0/463735299 >> 172.21.15.52:0/463735299 conn(0x7fd5e80f6530 msgr2=0x7fd5e80f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:44.625 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:44.622+0000 7fd5f00b8700 1 -- 172.21.15.52:0/463735299 shutdown_connections 2024-02-19T22:09:44.625 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:44.622+0000 7fd5f00b8700 1 -- 172.21.15.52:0/463735299 wait complete. 2024-02-19T22:09:45.092 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:44 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:09:43.606897+0000 mon.smithi052 (mon.0) 94 : cluster [DBG] mgrmap e14: smithi052.ghqrtk(active, starting, since 0.0525343s) 2024-02-19T22:09:45.093 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:44 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:43.609354+0000 mon.smithi052 (mon.0) 95 : audit [DBG] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix": "mon metadata", "id": "smithi052"}]: dispatch 2024-02-19T22:09:45.093 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:44 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:43.609991+0000 mon.smithi052 (mon.0) 96 : audit [DBG] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix": "mgr metadata", "who": "smithi052.ghqrtk", "id": "smithi052.ghqrtk"}]: dispatch 2024-02-19T22:09:45.093 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:44 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:43.610695+0000 mon.smithi052 (mon.0) 97 : audit [DBG] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T22:09:45.093 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:44 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:43.610891+0000 mon.smithi052 (mon.0) 98 : audit [DBG] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T22:09:45.093 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:44 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:43.611117+0000 mon.smithi052 (mon.0) 99 : audit [DBG] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T22:09:45.093 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:44 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:09:43.613019+0000 mon.smithi052 (mon.0) 100 : cluster [INF] Manager daemon smithi052.ghqrtk is now available 2024-02-19T22:09:45.093 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:44 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:43.717788+0000 mon.smithi052 (mon.0) 101 : audit [DBG] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T22:09:45.093 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:44 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:43.718994+0000 mon.smithi052 (mon.0) 102 : audit [DBG] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T22:09:45.093 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:44 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:43.742431+0000 mon.smithi052 (mon.0) 103 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi052.ghqrtk/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:09:45.093 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:44 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:43.746825+0000 mon.smithi052 (mon.0) 104 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi052.ghqrtk/trash_purge_schedule"}]: dispatch 2024-02-19T22:09:45.756 INFO:teuthology.orchestra.run.smithi052.stderr:mgr epoch 13 is available 2024-02-19T22:09:45.756 INFO:teuthology.orchestra.run.smithi052.stderr:Generating a dashboard self-signed certificate... 2024-02-19T22:09:45.756 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi052 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphvym4zi3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp6ou7w4ul:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2024-02-19T22:09:45.758 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:45 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:44.610955+0000 mgr.smithi052.ghqrtk (mgr.14162) 1 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-19T22:09:45.758 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:45 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:09:44.611280+0000 mon.smithi052 (mon.0) 105 : cluster [DBG] mgrmap e15: smithi052.ghqrtk(active, since 1.05692s) 2024-02-19T22:09:45.758 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:45 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:44.622764+0000 mgr.smithi052.ghqrtk (mgr.14162) 2 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-19T22:09:45.759 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:45 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:45.362260+0000 mon.smithi052 (mon.0) 106 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' 2024-02-19T22:09:46.491 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.383+0000 7fd1908cd700 1 Processor -- start 2024-02-19T22:09:46.491 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.383+0000 7fd1908cd700 1 -- start start 2024-02-19T22:09:46.491 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.383+0000 7fd1908cd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd18c072560 0x7fd18c070b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:46.492 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.383+0000 7fd1908cd700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd18c072980 con 0x7fd18c072560 2024-02-19T22:09:46.492 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.383+0000 7fd18ad9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd18c072560 0x7fd18c070b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:46.492 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.383+0000 7fd18ad9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd18c072560 0x7fd18c070b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:55430/0 (socket says 172.21.15.52:55430) 2024-02-19T22:09:46.492 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.383+0000 7fd18ad9d700 1 -- 172.21.15.52:0/2182572080 learned_addr learned my addr 172.21.15.52:0/2182572080 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:09:46.492 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.383+0000 7fd18ad9d700 1 -- 172.21.15.52:0/2182572080 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd18c070ff0 con 0x7fd18c072560 2024-02-19T22:09:46.492 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.383+0000 7fd18ad9d700 1 --2- 172.21.15.52:0/2182572080 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd18c072560 0x7fd18c070b00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fd17c00b5f0 tx=0x7fd17c006ee0).ready entity=mon.0 client_cookie=abb486bb0323605f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:46.492 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.384+0000 7fd189d9b700 1 -- 172.21.15.52:0/2182572080 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd17c00d600 con 0x7fd18c072560 2024-02-19T22:09:46.492 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.384+0000 7fd189d9b700 1 -- 172.21.15.52:0/2182572080 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd17c00dc30 con 0x7fd18c072560 2024-02-19T22:09:46.492 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.384+0000 7fd189d9b700 1 -- 172.21.15.52:0/2182572080 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd17c011d30 con 0x7fd18c072560 2024-02-19T22:09:46.492 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.384+0000 7fd1908cd700 1 -- 172.21.15.52:0/2182572080 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd18c072560 msgr2=0x7fd18c070b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:46.492 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.384+0000 7fd1908cd700 1 --2- 172.21.15.52:0/2182572080 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd18c072560 0x7fd18c070b00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fd17c00b5f0 tx=0x7fd17c006ee0).stop 2024-02-19T22:09:46.492 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.384+0000 7fd1908cd700 1 -- 172.21.15.52:0/2182572080 shutdown_connections 2024-02-19T22:09:46.492 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.384+0000 7fd1908cd700 1 --2- 172.21.15.52:0/2182572080 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd18c072560 0x7fd18c070b00 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:46.492 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.384+0000 7fd1908cd700 1 -- 172.21.15.52:0/2182572080 >> 172.21.15.52:0/2182572080 conn(0x7fd18c0f7510 msgr2=0x7fd18c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:46.493 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.384+0000 7fd1908cd700 1 -- 172.21.15.52:0/2182572080 shutdown_connections 2024-02-19T22:09:46.493 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.384+0000 7fd1908cd700 1 -- 172.21.15.52:0/2182572080 wait complete. 2024-02-19T22:09:46.493 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.385+0000 7fd1908cd700 1 Processor -- start 2024-02-19T22:09:46.493 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.385+0000 7fd1908cd700 1 -- start start 2024-02-19T22:09:46.493 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.385+0000 7fd1908cd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd18c072560 0x7fd18c110f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:46.493 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.385+0000 7fd1908cd700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd18c111480 con 0x7fd18c072560 2024-02-19T22:09:46.493 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.385+0000 7fd18ad9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd18c072560 0x7fd18c110f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:46.493 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.385+0000 7fd18ad9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd18c072560 0x7fd18c110f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:55446/0 (socket says 172.21.15.52:55446) 2024-02-19T22:09:46.493 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.385+0000 7fd18ad9d700 1 -- 172.21.15.52:0/3029294763 learned_addr learned my addr 172.21.15.52:0/3029294763 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:09:46.493 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.385+0000 7fd18ad9d700 1 -- 172.21.15.52:0/3029294763 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd17c00b200 con 0x7fd18c072560 2024-02-19T22:09:46.493 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.385+0000 7fd18ad9d700 1 --2- 172.21.15.52:0/3029294763 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd18c072560 0x7fd18c110f90 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fd17c00e040 tx=0x7fd17c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:46.493 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.386+0000 7fd1737fe700 1 -- 172.21.15.52:0/3029294763 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd17c00da60 con 0x7fd18c072560 2024-02-19T22:09:46.493 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.386+0000 7fd1737fe700 1 -- 172.21.15.52:0/3029294763 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd17c01b8e0 con 0x7fd18c072560 2024-02-19T22:09:46.493 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.386+0000 7fd1908cd700 1 -- 172.21.15.52:0/3029294763 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd18c111680 con 0x7fd18c072560 2024-02-19T22:09:46.493 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.386+0000 7fd1737fe700 1 -- 172.21.15.52:0/3029294763 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd17c01a900 con 0x7fd18c072560 2024-02-19T22:09:46.494 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.386+0000 7fd1908cd700 1 -- 172.21.15.52:0/3029294763 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd18c111b10 con 0x7fd18c072560 2024-02-19T22:09:46.494 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.386+0000 7fd1737fe700 1 -- 172.21.15.52:0/3029294763 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fd17c00f070 con 0x7fd18c072560 2024-02-19T22:09:46.494 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.387+0000 7fd1737fe700 1 --2- 172.21.15.52:0/3029294763 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd17802ffc0 0x7fd178032480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:46.494 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.387+0000 7fd1737fe700 1 -- 172.21.15.52:0/3029294763 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd17c041480 con 0x7fd18c072560 2024-02-19T22:09:46.494 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.387+0000 7fd18a59c700 1 --2- 172.21.15.52:0/3029294763 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd17802ffc0 0x7fd178032480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:46.494 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.387+0000 7fd18a59c700 1 --2- 172.21.15.52:0/3029294763 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd17802ffc0 0x7fd178032480 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fd1800096a0 tx=0x7fd180006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:46.494 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.387+0000 7fd1908cd700 1 -- 172.21.15.52:0/3029294763 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd18c045ad0 con 0x7fd18c072560 2024-02-19T22:09:46.494 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.391+0000 7fd1737fe700 1 -- 172.21.15.52:0/3029294763 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd17c016070 con 0x7fd18c072560 2024-02-19T22:09:46.566 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.564+0000 7fd1908cd700 1 -- 172.21.15.52:0/3029294763 --> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fd18c10ac00 con 0x7fd17802ffc0 2024-02-19T22:09:46.672 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.670+0000 7fd1737fe700 1 -- 172.21.15.52:0/3029294763 <== mgr.14162 v2:172.21.15.52:6800/539827537 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fd18c10ac00 con 0x7fd17802ffc0 2024-02-19T22:09:46.672 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: Self-signed certificate created 2024-02-19T22:09:46.674 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.672+0000 7fd1908cd700 1 -- 172.21.15.52:0/3029294763 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd17802ffc0 msgr2=0x7fd178032480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:46.675 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.672+0000 7fd1908cd700 1 --2- 172.21.15.52:0/3029294763 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd17802ffc0 0x7fd178032480 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fd1800096a0 tx=0x7fd180006b40).stop 2024-02-19T22:09:46.675 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.672+0000 7fd1908cd700 1 -- 172.21.15.52:0/3029294763 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd18c072560 msgr2=0x7fd18c110f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:46.675 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.672+0000 7fd1908cd700 1 --2- 172.21.15.52:0/3029294763 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd18c072560 0x7fd18c110f90 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fd17c00e040 tx=0x7fd17c006ee0).stop 2024-02-19T22:09:46.675 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.672+0000 7fd1908cd700 1 -- 172.21.15.52:0/3029294763 shutdown_connections 2024-02-19T22:09:46.675 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.672+0000 7fd1908cd700 1 --2- 172.21.15.52:0/3029294763 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd17802ffc0 0x7fd178032480 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:46.675 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.672+0000 7fd1908cd700 1 --2- 172.21.15.52:0/3029294763 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd18c072560 0x7fd18c110f90 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:46.675 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.672+0000 7fd1908cd700 1 -- 172.21.15.52:0/3029294763 >> 172.21.15.52:0/3029294763 conn(0x7fd18c0f7510 msgr2=0x7fd18c0f9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:46.675 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.672+0000 7fd1908cd700 1 -- 172.21.15.52:0/3029294763 shutdown_connections 2024-02-19T22:09:46.675 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:46.672+0000 7fd1908cd700 1 -- 172.21.15.52:0/3029294763 wait complete. 2024-02-19T22:09:47.462 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:47 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:09:46.363728+0000 mon.smithi052 (mon.0) 107 : cluster [DBG] mgrmap e16: smithi052.ghqrtk(active, since 2s) 2024-02-19T22:09:47.463 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:47 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:46.669835+0000 mon.smithi052 (mon.0) 108 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' 2024-02-19T22:09:47.463 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:47 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:46.671703+0000 mon.smithi052 (mon.0) 109 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' 2024-02-19T22:09:48.064 INFO:teuthology.orchestra.run.smithi052.stderr:Creating initial admin user... 2024-02-19T22:09:48.065 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi052 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphvym4zi3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp6ou7w4ul:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpfzcu7njx:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2024-02-19T22:09:48.842 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:48 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:46.566511+0000 mgr.smithi052.ghqrtk (mgr.14162) 3 : audit [DBG] from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:09:48.951 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:48.710+0000 7fa602350700 1 Processor -- start 2024-02-19T22:09:48.951 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:48.710+0000 7fa602350700 1 -- start start 2024-02-19T22:09:48.951 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:48.710+0000 7fa602350700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5fc0703d0 0x7fa5fc0feab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:48.952 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:48.710+0000 7fa602350700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5fc0fefa0 con 0x7fa5fc0703d0 2024-02-19T22:09:48.952 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:48.711+0000 7fa5fbfff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5fc0703d0 0x7fa5fc0feab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:48.952 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:48.711+0000 7fa5fbfff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5fc0703d0 0x7fa5fc0feab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:55448/0 (socket says 172.21.15.52:55448) 2024-02-19T22:09:48.952 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:48.711+0000 7fa5fbfff700 1 -- 172.21.15.52:0/1580742093 learned_addr learned my addr 172.21.15.52:0/1580742093 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:09:48.952 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:48.711+0000 7fa5fbfff700 1 -- 172.21.15.52:0/1580742093 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5fc0ff0e0 con 0x7fa5fc0703d0 2024-02-19T22:09:48.952 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:48.711+0000 7fa5fbfff700 1 --2- 172.21.15.52:0/1580742093 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5fc0703d0 0x7fa5fc0feab0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fa5ec00b5f0 tx=0x7fa5ec006ee0).ready entity=mon.0 client_cookie=86f3caf0c35410fd server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:48.952 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:48.711+0000 7fa5faffd700 1 -- 172.21.15.52:0/1580742093 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5ec00d600 con 0x7fa5fc0703d0 2024-02-19T22:09:48.952 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:48.711+0000 7fa5faffd700 1 -- 172.21.15.52:0/1580742093 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa5ec00dc30 con 0x7fa5fc0703d0 2024-02-19T22:09:48.952 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:48.711+0000 7fa5faffd700 1 -- 172.21.15.52:0/1580742093 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5ec011d30 con 0x7fa5fc0703d0 2024-02-19T22:09:48.952 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:48.712+0000 7fa602350700 1 -- 172.21.15.52:0/1580742093 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5fc0703d0 msgr2=0x7fa5fc0feab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:48.952 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:48.712+0000 7fa602350700 1 --2- 172.21.15.52:0/1580742093 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5fc0703d0 0x7fa5fc0feab0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fa5ec00b5f0 tx=0x7fa5ec006ee0).stop 2024-02-19T22:09:48.952 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:48.712+0000 7fa602350700 1 -- 172.21.15.52:0/1580742093 shutdown_connections 2024-02-19T22:09:48.952 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:48.712+0000 7fa602350700 1 --2- 172.21.15.52:0/1580742093 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5fc0703d0 0x7fa5fc0feab0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:48.952 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:48.712+0000 7fa602350700 1 -- 172.21.15.52:0/1580742093 >> 172.21.15.52:0/1580742093 conn(0x7fa5fc0f6530 msgr2=0x7fa5fc0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:48.953 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:48.712+0000 7fa602350700 1 -- 172.21.15.52:0/1580742093 shutdown_connections 2024-02-19T22:09:48.953 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:48.712+0000 7fa602350700 1 -- 172.21.15.52:0/1580742093 wait complete. 2024-02-19T22:09:48.953 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:48.712+0000 7fa602350700 1 Processor -- start 2024-02-19T22:09:48.953 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:48.713+0000 7fa602350700 1 -- start start 2024-02-19T22:09:48.953 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:48.713+0000 7fa602350700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5fc0703d0 0x7fa5fc10ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:48.953 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:48.713+0000 7fa602350700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5fc10b0f0 con 0x7fa5fc0703d0 2024-02-19T22:09:48.953 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:48.713+0000 7fa5fbfff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5fc0703d0 0x7fa5fc10ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:48.953 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:48.713+0000 7fa5fbfff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5fc0703d0 0x7fa5fc10ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:55452/0 (socket says 172.21.15.52:55452) 2024-02-19T22:09:48.953 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:48.713+0000 7fa5fbfff700 1 -- 172.21.15.52:0/476097587 learned_addr learned my addr 172.21.15.52:0/476097587 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:09:48.953 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:48.713+0000 7fa5fbfff700 1 -- 172.21.15.52:0/476097587 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5ec00b200 con 0x7fa5fc0703d0 2024-02-19T22:09:48.953 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:48.713+0000 7fa5fbfff700 1 --2- 172.21.15.52:0/476097587 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5fc0703d0 0x7fa5fc10ac00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fa5ec00c010 tx=0x7fa5ec006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:48.953 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:48.713+0000 7fa5f8ff9700 1 -- 172.21.15.52:0/476097587 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5ec00da60 con 0x7fa5fc0703d0 2024-02-19T22:09:48.953 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:48.713+0000 7fa5f8ff9700 1 -- 172.21.15.52:0/476097587 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa5ec01b8e0 con 0x7fa5fc0703d0 2024-02-19T22:09:48.953 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:48.713+0000 7fa602350700 1 -- 172.21.15.52:0/476097587 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5fc10b2f0 con 0x7fa5fc0703d0 2024-02-19T22:09:48.953 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:48.714+0000 7fa5f8ff9700 1 -- 172.21.15.52:0/476097587 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5ec01a990 con 0x7fa5fc0703d0 2024-02-19T22:09:48.953 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:48.714+0000 7fa602350700 1 -- 172.21.15.52:0/476097587 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5fc10b780 con 0x7fa5fc0703d0 2024-02-19T22:09:48.954 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:48.714+0000 7fa5f8ff9700 1 -- 172.21.15.52:0/476097587 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fa5ec00f070 con 0x7fa5fc0703d0 2024-02-19T22:09:48.954 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:48.714+0000 7fa5f8ff9700 1 --2- 172.21.15.52:0/476097587 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa5e802ffc0 0x7fa5e8032480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:48.954 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:48.714+0000 7fa5f8ff9700 1 -- 172.21.15.52:0/476097587 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa5ec041520 con 0x7fa5fc0703d0 2024-02-19T22:09:48.954 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:48.715+0000 7fa5fb7fe700 1 --2- 172.21.15.52:0/476097587 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa5e802ffc0 0x7fa5e8032480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:48.954 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:48.715+0000 7fa602350700 1 -- 172.21.15.52:0/476097587 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5fc045ad0 con 0x7fa5fc0703d0 2024-02-19T22:09:48.954 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:48.715+0000 7fa5fb7fe700 1 --2- 172.21.15.52:0/476097587 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa5e802ffc0 0x7fa5e8032480 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fa5f00096a0 tx=0x7fa5f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:48.954 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:48.719+0000 7fa5f8ff9700 1 -- 172.21.15.52:0/476097587 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa5ec016070 con 0x7fa5fc0703d0 2024-02-19T22:09:48.954 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:48.896+0000 7fa602350700 1 -- 172.21.15.52:0/476097587 --> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fa5fc058f20 con 0x7fa5e802ffc0 2024-02-19T22:09:49.141 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:49.140+0000 7fa5f8ff9700 1 -- 172.21.15.52:0/476097587 <== mgr.14162 v2:172.21.15.52:6800/539827537 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7fa5fc058f20 con 0x7fa5e802ffc0 2024-02-19T22:09:49.141 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$1TZeHbf2to2LZq8obDGJMeY9gxaOtx.stugdy1GvgjewiqvQwbW5m", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1708380589, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-19T22:09:49.144 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:49.142+0000 7fa602350700 1 -- 172.21.15.52:0/476097587 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa5e802ffc0 msgr2=0x7fa5e8032480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:49.144 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:49.142+0000 7fa602350700 1 --2- 172.21.15.52:0/476097587 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa5e802ffc0 0x7fa5e8032480 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fa5f00096a0 tx=0x7fa5f0006b40).stop 2024-02-19T22:09:49.144 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:49.142+0000 7fa602350700 1 -- 172.21.15.52:0/476097587 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5fc0703d0 msgr2=0x7fa5fc10ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:49.144 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:49.142+0000 7fa602350700 1 --2- 172.21.15.52:0/476097587 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5fc0703d0 0x7fa5fc10ac00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fa5ec00c010 tx=0x7fa5ec006ee0).stop 2024-02-19T22:09:49.144 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:49.142+0000 7fa602350700 1 -- 172.21.15.52:0/476097587 shutdown_connections 2024-02-19T22:09:49.144 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:49.142+0000 7fa602350700 1 --2- 172.21.15.52:0/476097587 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa5e802ffc0 0x7fa5e8032480 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:49.145 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:49.142+0000 7fa602350700 1 --2- 172.21.15.52:0/476097587 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5fc0703d0 0x7fa5fc10ac00 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:49.145 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:49.142+0000 7fa602350700 1 -- 172.21.15.52:0/476097587 >> 172.21.15.52:0/476097587 conn(0x7fa5fc0f6530 msgr2=0x7fa5fc0f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:49.145 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:49.142+0000 7fa602350700 1 -- 172.21.15.52:0/476097587 shutdown_connections 2024-02-19T22:09:49.145 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:49.142+0000 7fa602350700 1 -- 172.21.15.52:0/476097587 wait complete. 2024-02-19T22:09:49.865 INFO:teuthology.orchestra.run.smithi052.stderr:Fetching dashboard port number... 2024-02-19T22:09:49.865 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi052 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphvym4zi3:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp6ou7w4ul:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2024-02-19T22:09:50.469 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.460+0000 7f522d593700 1 Processor -- start 2024-02-19T22:09:50.469 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.460+0000 7f522d593700 1 -- start start 2024-02-19T22:09:50.469 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.460+0000 7f522d593700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5228072540 0x7f5228070ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:50.469 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.460+0000 7f522d593700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5228072960 con 0x7f5228072540 2024-02-19T22:09:50.469 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.460+0000 7f5226ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5228072540 0x7f5228070ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:50.469 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.460+0000 7f5226ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5228072540 0x7f5228070ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:55468/0 (socket says 172.21.15.52:55468) 2024-02-19T22:09:50.470 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.460+0000 7f5226ffd700 1 -- 172.21.15.52:0/94835419 learned_addr learned my addr 172.21.15.52:0/94835419 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:09:50.470 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.461+0000 7f5226ffd700 1 -- 172.21.15.52:0/94835419 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5228070fd0 con 0x7f5228072540 2024-02-19T22:09:50.470 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.461+0000 7f5226ffd700 1 --2- 172.21.15.52:0/94835419 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5228072540 0x7f5228070ae0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f521000b5f0 tx=0x7f5210006ee0).ready entity=mon.0 client_cookie=e1f17ff7e44fe7e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:50.470 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.461+0000 7f5225ffb700 1 -- 172.21.15.52:0/94835419 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f521000d600 con 0x7f5228072540 2024-02-19T22:09:50.470 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.461+0000 7f5225ffb700 1 -- 172.21.15.52:0/94835419 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f521000dc30 con 0x7f5228072540 2024-02-19T22:09:50.470 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.461+0000 7f5225ffb700 1 -- 172.21.15.52:0/94835419 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5210011d30 con 0x7f5228072540 2024-02-19T22:09:50.470 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.461+0000 7f522d593700 1 -- 172.21.15.52:0/94835419 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5228072540 msgr2=0x7f5228070ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:50.470 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.461+0000 7f522d593700 1 --2- 172.21.15.52:0/94835419 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5228072540 0x7f5228070ae0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f521000b5f0 tx=0x7f5210006ee0).stop 2024-02-19T22:09:50.470 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.461+0000 7f522d593700 1 -- 172.21.15.52:0/94835419 shutdown_connections 2024-02-19T22:09:50.470 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.462+0000 7f522d593700 1 --2- 172.21.15.52:0/94835419 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5228072540 0x7f5228070ae0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:50.470 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.462+0000 7f522d593700 1 -- 172.21.15.52:0/94835419 >> 172.21.15.52:0/94835419 conn(0x7f52280f76a0 msgr2=0x7f52280f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:50.470 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.462+0000 7f522d593700 1 -- 172.21.15.52:0/94835419 shutdown_connections 2024-02-19T22:09:50.470 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.462+0000 7f522d593700 1 -- 172.21.15.52:0/94835419 wait complete. 2024-02-19T22:09:50.470 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.462+0000 7f522d593700 1 Processor -- start 2024-02-19T22:09:50.470 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.462+0000 7f522d593700 1 -- start start 2024-02-19T22:09:50.470 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.463+0000 7f522d593700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5228072540 0x7f522810ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:50.471 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.463+0000 7f522d593700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f522810d350 con 0x7f5228072540 2024-02-19T22:09:50.471 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.463+0000 7f5226ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5228072540 0x7f522810ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:50.471 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.463+0000 7f5226ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5228072540 0x7f522810ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:55478/0 (socket says 172.21.15.52:55478) 2024-02-19T22:09:50.471 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.463+0000 7f5226ffd700 1 -- 172.21.15.52:0/3903664281 learned_addr learned my addr 172.21.15.52:0/3903664281 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:09:50.471 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.463+0000 7f5226ffd700 1 -- 172.21.15.52:0/3903664281 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f521000b200 con 0x7f5228072540 2024-02-19T22:09:50.471 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.463+0000 7f5226ffd700 1 --2- 172.21.15.52:0/3903664281 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5228072540 0x7f522810ce60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f521000b1b0 tx=0x7f5210006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:50.471 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.463+0000 7f521f7fe700 1 -- 172.21.15.52:0/3903664281 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f521000d960 con 0x7f5228072540 2024-02-19T22:09:50.471 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.463+0000 7f521f7fe700 1 -- 172.21.15.52:0/3903664281 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f521000dac0 con 0x7f5228072540 2024-02-19T22:09:50.471 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.463+0000 7f522d593700 1 -- 172.21.15.52:0/3903664281 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f522810d550 con 0x7f5228072540 2024-02-19T22:09:50.471 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.463+0000 7f521f7fe700 1 -- 172.21.15.52:0/3903664281 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f521001a7c0 con 0x7f5228072540 2024-02-19T22:09:50.471 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.463+0000 7f522d593700 1 -- 172.21.15.52:0/3903664281 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f522810e160 con 0x7f5228072540 2024-02-19T22:09:50.471 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.464+0000 7f521f7fe700 1 -- 172.21.15.52:0/3903664281 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f521000f070 con 0x7f5228072540 2024-02-19T22:09:50.471 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.464+0000 7f521f7fe700 1 --2- 172.21.15.52:0/3903664281 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f5208030070 0x7f5208032530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:50.471 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.464+0000 7f521f7fe700 1 -- 172.21.15.52:0/3903664281 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5210041440 con 0x7f5228072540 2024-02-19T22:09:50.471 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.465+0000 7f522d593700 1 -- 172.21.15.52:0/3903664281 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5228059100 con 0x7f5228072540 2024-02-19T22:09:50.471 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.465+0000 7f52267fc700 1 --2- 172.21.15.52:0/3903664281 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f5208030070 0x7f5208032530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:50.472 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.465+0000 7f52267fc700 1 --2- 172.21.15.52:0/3903664281 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f5208030070 0x7f5208032530 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f52180096a0 tx=0x7f5218006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:50.472 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.469+0000 7f521f7fe700 1 -- 172.21.15.52:0/3903664281 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5210016070 con 0x7f5228072540 2024-02-19T22:09:50.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:50 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:48.897815+0000 mgr.smithi052.ghqrtk (mgr.14162) 4 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:09:50.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:50 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:49.141235+0000 mon.smithi052 (mon.0) 110 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' 2024-02-19T22:09:50.630 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.628+0000 7f522d593700 1 -- 172.21.15.52:0/3903664281 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f5228045ad0 con 0x7f5228072540 2024-02-19T22:09:50.630 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.629+0000 7f521f7fe700 1 -- 172.21.15.52:0/3903664281 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v10) v1 ==== 112+0+5 (secure 0 0 0) 0x7f5210025460 con 0x7f5228072540 2024-02-19T22:09:50.630 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 8443 2024-02-19T22:09:50.633 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.631+0000 7f522d593700 1 -- 172.21.15.52:0/3903664281 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f5208030070 msgr2=0x7f5208032530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:50.633 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.631+0000 7f522d593700 1 --2- 172.21.15.52:0/3903664281 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f5208030070 0x7f5208032530 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f52180096a0 tx=0x7f5218006b40).stop 2024-02-19T22:09:50.633 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.631+0000 7f522d593700 1 -- 172.21.15.52:0/3903664281 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5228072540 msgr2=0x7f522810ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:50.633 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.631+0000 7f522d593700 1 --2- 172.21.15.52:0/3903664281 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5228072540 0x7f522810ce60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f521000b1b0 tx=0x7f5210006ee0).stop 2024-02-19T22:09:50.633 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.631+0000 7f522d593700 1 -- 172.21.15.52:0/3903664281 shutdown_connections 2024-02-19T22:09:50.633 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.631+0000 7f522d593700 1 --2- 172.21.15.52:0/3903664281 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f5208030070 0x7f5208032530 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:50.633 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.631+0000 7f522d593700 1 --2- 172.21.15.52:0/3903664281 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5228072540 0x7f522810ce60 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:50.633 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.631+0000 7f522d593700 1 -- 172.21.15.52:0/3903664281 >> 172.21.15.52:0/3903664281 conn(0x7f52280f76a0 msgr2=0x7f52280f90b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:50.633 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.631+0000 7f522d593700 1 -- 172.21.15.52:0/3903664281 shutdown_connections 2024-02-19T22:09:50.633 INFO:teuthology.orchestra.run.smithi052.stderr:/usr/bin/ceph: 2024-02-19T22:09:50.631+0000 7f522d593700 1 -- 172.21.15.52:0/3903664281 wait complete. 2024-02-19T22:09:51.257 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-19T22:09:51.260 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:51 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:09:50.145741+0000 mon.smithi052 (mon.0) 111 : cluster [DBG] mgrmap e17: smithi052.ghqrtk(active, since 6s) 2024-02-19T22:09:51.260 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:51 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:50.630341+0000 mon.smithi052 (mon.0) 112 : audit [DBG] from='client.? 172.21.15.52:0/3903664281' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-19T22:09:51.267 INFO:teuthology.orchestra.run.smithi052.stderr:systemctl: disabled 2024-02-19T22:09:51.268 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: systemctl is-active firewalld.service 2024-02-19T22:09:51.276 INFO:teuthology.orchestra.run.smithi052.stderr:systemctl: inactive 2024-02-19T22:09:51.277 INFO:teuthology.orchestra.run.smithi052.stderr:firewalld.service is not enabled 2024-02-19T22:09:51.277 INFO:teuthology.orchestra.run.smithi052.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-19T22:09:51.280 INFO:teuthology.orchestra.run.smithi052.stderr:Ceph Dashboard is now available at: 2024-02-19T22:09:51.280 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-02-19T22:09:51.280 INFO:teuthology.orchestra.run.smithi052.stderr: URL: https://smithi052.front.sepia.ceph.com:8443/ 2024-02-19T22:09:51.280 INFO:teuthology.orchestra.run.smithi052.stderr: User: admin 2024-02-19T22:09:51.281 INFO:teuthology.orchestra.run.smithi052.stderr: Password: cd7622zwxs 2024-02-19T22:09:51.281 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-02-19T22:09:51.281 INFO:teuthology.orchestra.run.smithi052.stderr:You can access the Ceph CLI with: 2024-02-19T22:09:51.281 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-02-19T22:09:51.281 INFO:teuthology.orchestra.run.smithi052.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-19T22:09:51.281 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-02-19T22:09:51.281 INFO:teuthology.orchestra.run.smithi052.stderr:Please consider enabling telemetry to help improve Ceph: 2024-02-19T22:09:51.281 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-02-19T22:09:51.281 INFO:teuthology.orchestra.run.smithi052.stderr: ceph telemetry on 2024-02-19T22:09:51.281 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-02-19T22:09:51.281 INFO:teuthology.orchestra.run.smithi052.stderr:For more information see: 2024-02-19T22:09:51.281 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-02-19T22:09:51.281 INFO:teuthology.orchestra.run.smithi052.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2024-02-19T22:09:51.282 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-02-19T22:09:51.282 INFO:teuthology.orchestra.run.smithi052.stderr:Bootstrap complete. 2024-02-19T22:09:51.282 INFO:teuthology.orchestra.run.smithi052.stderr:Releasing lock 140610839112168 on /run/cephadm/56739a54-cf73-11ee-95bb-87774f69a715.lock 2024-02-19T22:09:51.282 INFO:teuthology.orchestra.run.smithi052.stderr:Lock 140610839112168 released on /run/cephadm/56739a54-cf73-11ee-95bb-87774f69a715.lock 2024-02-19T22:09:51.320 INFO:tasks.cephadm:Fetching config... 2024-02-19T22:09:51.320 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-02-19T22:09:51.320 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-19T22:09:51.380 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-19T22:09:51.381 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-02-19T22:09:51.381 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-19T22:09:51.439 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-19T22:09:51.439 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-02-19T22:09:51.439 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/mon.smithi052/keyring of=/dev/stdout 2024-02-19T22:09:51.509 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-19T22:09:51.509 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-02-19T22:09:51.509 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-19T22:09:51.567 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-19T22:09:51.568 DEBUG:teuthology.orchestra.run.smithi052:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCeHpDwSV4bWEN4DlspoLjwZAe49WSN4rYbgGQ2aTR1gtmwj5Vpfk2vL8OXNpwOCCYDZTUwvmxVeUgmBG3NMzOK9lVaiOuKcX6prFa5AHxLEg8FMo/4uuiAodKtCEL75Q6JPZ6GPhs06eW4jc7zF8eWjSdSapepUg04jNzjR0O6ZX2f3MXjLP3JZKnhb/gRKNTVWvGa5dB7OwsRwCL96xCZs6tczrHLhZxlshrsTsvS8WVTNpVqBy8jdsbgHdO33mxVDNkgHBQaU3te2qxtovycnI0l84SvtbnMtDDsBQeOecVD0IiBHdk99yEHfNiQhVoe7A/dG89DQnnbxsEU1VkX9gMybPE1pXFTWrAJzFsnXDg+iT+Q7KecCap6WzucNgFNUq82TYfpS9QPijEJ2qw90KnwogdD9E5dKtW7XxqkPbFJ2/bONmR6sQYpUeWyXkDJohAaZdQlKKys0Gv1Pcab80zIuLKAh2/cqgM2rZTnulwgM0CxHGsiokUA2I3U5+c= ceph-56739a54-cf73-11ee-95bb-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-19T22:09:51.648 INFO:teuthology.orchestra.run.smithi052.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCeHpDwSV4bWEN4DlspoLjwZAe49WSN4rYbgGQ2aTR1gtmwj5Vpfk2vL8OXNpwOCCYDZTUwvmxVeUgmBG3NMzOK9lVaiOuKcX6prFa5AHxLEg8FMo/4uuiAodKtCEL75Q6JPZ6GPhs06eW4jc7zF8eWjSdSapepUg04jNzjR0O6ZX2f3MXjLP3JZKnhb/gRKNTVWvGa5dB7OwsRwCL96xCZs6tczrHLhZxlshrsTsvS8WVTNpVqBy8jdsbgHdO33mxVDNkgHBQaU3te2qxtovycnI0l84SvtbnMtDDsBQeOecVD0IiBHdk99yEHfNiQhVoe7A/dG89DQnnbxsEU1VkX9gMybPE1pXFTWrAJzFsnXDg+iT+Q7KecCap6WzucNgFNUq82TYfpS9QPijEJ2qw90KnwogdD9E5dKtW7XxqkPbFJ2/bONmR6sQYpUeWyXkDJohAaZdQlKKys0Gv1Pcab80zIuLKAh2/cqgM2rZTnulwgM0CxHGsiokUA2I3U5+c= ceph-56739a54-cf73-11ee-95bb-87774f69a715 2024-02-19T22:09:51.660 DEBUG:teuthology.orchestra.run.smithi088:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCeHpDwSV4bWEN4DlspoLjwZAe49WSN4rYbgGQ2aTR1gtmwj5Vpfk2vL8OXNpwOCCYDZTUwvmxVeUgmBG3NMzOK9lVaiOuKcX6prFa5AHxLEg8FMo/4uuiAodKtCEL75Q6JPZ6GPhs06eW4jc7zF8eWjSdSapepUg04jNzjR0O6ZX2f3MXjLP3JZKnhb/gRKNTVWvGa5dB7OwsRwCL96xCZs6tczrHLhZxlshrsTsvS8WVTNpVqBy8jdsbgHdO33mxVDNkgHBQaU3te2qxtovycnI0l84SvtbnMtDDsBQeOecVD0IiBHdk99yEHfNiQhVoe7A/dG89DQnnbxsEU1VkX9gMybPE1pXFTWrAJzFsnXDg+iT+Q7KecCap6WzucNgFNUq82TYfpS9QPijEJ2qw90KnwogdD9E5dKtW7XxqkPbFJ2/bONmR6sQYpUeWyXkDJohAaZdQlKKys0Gv1Pcab80zIuLKAh2/cqgM2rZTnulwgM0CxHGsiokUA2I3U5+c= ceph-56739a54-cf73-11ee-95bb-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-19T22:09:51.707 INFO:teuthology.orchestra.run.smithi088.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCeHpDwSV4bWEN4DlspoLjwZAe49WSN4rYbgGQ2aTR1gtmwj5Vpfk2vL8OXNpwOCCYDZTUwvmxVeUgmBG3NMzOK9lVaiOuKcX6prFa5AHxLEg8FMo/4uuiAodKtCEL75Q6JPZ6GPhs06eW4jc7zF8eWjSdSapepUg04jNzjR0O6ZX2f3MXjLP3JZKnhb/gRKNTVWvGa5dB7OwsRwCL96xCZs6tczrHLhZxlshrsTsvS8WVTNpVqBy8jdsbgHdO33mxVDNkgHBQaU3te2qxtovycnI0l84SvtbnMtDDsBQeOecVD0IiBHdk99yEHfNiQhVoe7A/dG89DQnnbxsEU1VkX9gMybPE1pXFTWrAJzFsnXDg+iT+Q7KecCap6WzucNgFNUq82TYfpS9QPijEJ2qw90KnwogdD9E5dKtW7XxqkPbFJ2/bONmR6sQYpUeWyXkDJohAaZdQlKKys0Gv1Pcab80zIuLKAh2/cqgM2rZTnulwgM0CxHGsiokUA2I3U5+c= ceph-56739a54-cf73-11ee-95bb-87774f69a715 2024-02-19T22:09:51.719 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-19T22:09:55.110 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:09:55.107+0000 7fba7d727700 1 -- 172.21.15.52:0/876062997 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fba780f5410 msgr2=0x7fba780f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:55.110 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:09:55.107+0000 7fba7d727700 1 --2- 172.21.15.52:0/876062997 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fba780f5410 0x7fba780f5830 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fba70009fb0 tx=0x7fba70009b30).stop 2024-02-19T22:09:55.110 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:09:55.107+0000 7fba7d727700 1 -- 172.21.15.52:0/876062997 shutdown_connections 2024-02-19T22:09:55.110 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:09:55.107+0000 7fba7d727700 1 --2- 172.21.15.52:0/876062997 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fba780f5410 0x7fba780f5830 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:55.110 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:09:55.107+0000 7fba7d727700 1 -- 172.21.15.52:0/876062997 >> 172.21.15.52:0/876062997 conn(0x7fba780f0b70 msgr2=0x7fba780f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:55.110 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:09:55.107+0000 7fba7d727700 1 -- 172.21.15.52:0/876062997 shutdown_connections 2024-02-19T22:09:55.110 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:09:55.107+0000 7fba7d727700 1 -- 172.21.15.52:0/876062997 wait complete. 2024-02-19T22:09:55.110 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:09:55.108+0000 7fba7d727700 1 Processor -- start 2024-02-19T22:09:55.110 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:09:55.108+0000 7fba7d727700 1 -- start start 2024-02-19T22:09:55.110 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:09:55.108+0000 7fba7d727700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fba780f5410 0x7fba781056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:55.111 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:09:55.108+0000 7fba7d727700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba78105bd0 con 0x7fba780f5410 2024-02-19T22:09:55.111 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:09:55.108+0000 7fba76ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fba780f5410 0x7fba781056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:55.111 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:09:55.108+0000 7fba76ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fba780f5410 0x7fba781056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:39400/0 (socket says 172.21.15.52:39400) 2024-02-19T22:09:55.111 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:09:55.108+0000 7fba76ffd700 1 -- 172.21.15.52:0/3339934527 learned_addr learned my addr 172.21.15.52:0/3339934527 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:09:55.111 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:09:55.108+0000 7fba76ffd700 1 -- 172.21.15.52:0/3339934527 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba70005040 con 0x7fba780f5410 2024-02-19T22:09:55.111 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:09:55.108+0000 7fba76ffd700 1 --2- 172.21.15.52:0/3339934527 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fba780f5410 0x7fba781056e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fba70005010 tx=0x7fba7000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:55.111 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:09:55.108+0000 7fba6f7fe700 1 -- 172.21.15.52:0/3339934527 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba70009320 con 0x7fba780f5410 2024-02-19T22:09:55.111 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:09:55.108+0000 7fba6f7fe700 1 -- 172.21.15.52:0/3339934527 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fba70013380 con 0x7fba780f5410 2024-02-19T22:09:55.111 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:09:55.108+0000 7fba7d727700 1 -- 172.21.15.52:0/3339934527 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba78105dd0 con 0x7fba780f5410 2024-02-19T22:09:55.111 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:09:55.109+0000 7fba6f7fe700 1 -- 172.21.15.52:0/3339934527 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba70014e50 con 0x7fba780f5410 2024-02-19T22:09:55.111 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:09:55.109+0000 7fba7d727700 1 -- 172.21.15.52:0/3339934527 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba78106260 con 0x7fba780f5410 2024-02-19T22:09:55.112 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:09:55.109+0000 7fba6f7fe700 1 -- 172.21.15.52:0/3339934527 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fba700263a0 con 0x7fba780f5410 2024-02-19T22:09:55.112 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:09:55.109+0000 7fba7d727700 1 -- 172.21.15.52:0/3339934527 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba78045ad0 con 0x7fba780f5410 2024-02-19T22:09:55.112 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:09:55.109+0000 7fba6f7fe700 1 --2- 172.21.15.52:0/3339934527 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fba6402ff10 0x7fba640323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:55.112 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:09:55.110+0000 7fba6f7fe700 1 -- 172.21.15.52:0/3339934527 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fba70041180 con 0x7fba780f5410 2024-02-19T22:09:55.112 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:09:55.110+0000 7fba767fc700 1 --2- 172.21.15.52:0/3339934527 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fba6402ff10 0x7fba640323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:55.112 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:09:55.110+0000 7fba767fc700 1 --2- 172.21.15.52:0/3339934527 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fba6402ff10 0x7fba640323d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fba600096a0 tx=0x7fba60006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:55.115 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:09:55.114+0000 7fba6f7fe700 1 -- 172.21.15.52:0/3339934527 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fba7001d470 con 0x7fba780f5410 2024-02-19T22:09:55.277 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:09:55.276+0000 7fba7d727700 1 -- 172.21.15.52:0/3339934527 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fba780fafa0 con 0x7fba780f5410 2024-02-19T22:09:55.280 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:09:55.279+0000 7fba6f7fe700 1 -- 172.21.15.52:0/3339934527 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v11)=0 v11) v1 ==== 125+0+0 (secure 0 0 0) 0x7fba70014420 con 0x7fba780f5410 2024-02-19T22:09:55.282 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:09:55.280+0000 7fba7d727700 1 -- 172.21.15.52:0/3339934527 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fba6402ff10 msgr2=0x7fba640323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:55.282 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:09:55.280+0000 7fba7d727700 1 --2- 172.21.15.52:0/3339934527 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fba6402ff10 0x7fba640323d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fba600096a0 tx=0x7fba60006b40).stop 2024-02-19T22:09:55.282 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:09:55.280+0000 7fba7d727700 1 -- 172.21.15.52:0/3339934527 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fba780f5410 msgr2=0x7fba781056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:55.282 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:09:55.280+0000 7fba7d727700 1 --2- 172.21.15.52:0/3339934527 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fba780f5410 0x7fba781056e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fba70005010 tx=0x7fba7000b040).stop 2024-02-19T22:09:55.282 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:09:55.281+0000 7fba7d727700 1 -- 172.21.15.52:0/3339934527 shutdown_connections 2024-02-19T22:09:55.282 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:09:55.281+0000 7fba7d727700 1 --2- 172.21.15.52:0/3339934527 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fba6402ff10 0x7fba640323d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:55.282 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:09:55.281+0000 7fba7d727700 1 --2- 172.21.15.52:0/3339934527 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fba780f5410 0x7fba781056e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:55.282 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:09:55.281+0000 7fba7d727700 1 -- 172.21.15.52:0/3339934527 >> 172.21.15.52:0/3339934527 conn(0x7fba780f0b70 msgr2=0x7fba780f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:55.282 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:09:55.281+0000 7fba7d727700 1 -- 172.21.15.52:0/3339934527 shutdown_connections 2024-02-19T22:09:55.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:09:55.281+0000 7fba7d727700 1 -- 172.21.15.52:0/3339934527 wait complete. 2024-02-19T22:09:56.469 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi088 2024-02-19T22:09:56.470 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-02-19T22:09:56.470 DEBUG:teuthology.orchestra.run.smithi088:> dd of=/etc/ceph/ceph.conf 2024-02-19T22:09:56.492 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-02-19T22:09:56.492 DEBUG:teuthology.orchestra.run.smithi088:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-19T22:09:56.554 INFO:tasks.cephadm:Adding host smithi088 to orchestrator... 2024-02-19T22:09:56.554 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph orch host add smithi088 2024-02-19T22:09:56.591 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:55.280305+0000 mon.smithi052 (mon.0) 113 : audit [INF] from='client.? 172.21.15.52:0/3339934527' entity='client.admin' 2024-02-19T22:09:57.842 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:57 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:56.486315+0000 mon.smithi052 (mon.0) 114 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' 2024-02-19T22:09:57.842 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:57 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:09:56.779444+0000 mon.smithi052 (mon.0) 115 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' 2024-02-19T22:09:57.842 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:09:57 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cephadm 2024-02-19T22:09:56.785991+0000 mgr.smithi052.ghqrtk (mgr.14162) 5 : cephadm [INF] Deploying daemon alertmanager.smithi052 on smithi052 2024-02-19T22:10:05.342 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:10:05 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:10:03.612151+0000 mgr.smithi052.ghqrtk (mgr.14162) 6 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:07.342 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:10:07 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:10:05.612649+0000 mgr.smithi052.ghqrtk (mgr.14162) 7 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:09.342 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:10:09 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:10:07.613091+0000 mgr.smithi052.ghqrtk (mgr.14162) 8 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:11.342 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:10:11 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:10:09.613342+0000 mgr.smithi052.ghqrtk (mgr.14162) 9 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:13.342 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:10:13 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:10:11.613707+0000 mgr.smithi052.ghqrtk (mgr.14162) 10 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:15.342 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:10:15 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:10:13.613932+0000 mgr.smithi052.ghqrtk (mgr.14162) 11 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:17.342 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:10:17 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:10:15.614361+0000 mgr.smithi052.ghqrtk (mgr.14162) 12 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:18.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:18.496+0000 7fdc3eb4e700 1 -- 172.21.15.88:0/473539680 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdc380f6510 msgr2=0x7fdc380f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:18.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:18.496+0000 7fdc3eb4e700 1 --2- 172.21.15.88:0/473539680 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdc380f6510 0x7fdc380f6930 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fdc2c009fb0 tx=0x7fdc2c009b30).stop 2024-02-19T22:10:18.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:18.496+0000 7fdc3eb4e700 1 -- 172.21.15.88:0/473539680 shutdown_connections 2024-02-19T22:10:18.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:18.496+0000 7fdc3eb4e700 1 --2- 172.21.15.88:0/473539680 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdc380f6510 0x7fdc380f6930 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:18.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:18.496+0000 7fdc3eb4e700 1 -- 172.21.15.88:0/473539680 >> 172.21.15.88:0/473539680 conn(0x7fdc380f1c70 msgr2=0x7fdc380f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:18.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:18.496+0000 7fdc3eb4e700 1 -- 172.21.15.88:0/473539680 shutdown_connections 2024-02-19T22:10:18.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:18.496+0000 7fdc3eb4e700 1 -- 172.21.15.88:0/473539680 wait complete. 2024-02-19T22:10:18.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:18.497+0000 7fdc3eb4e700 1 Processor -- start 2024-02-19T22:10:18.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:18.497+0000 7fdc3eb4e700 1 -- start start 2024-02-19T22:10:18.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:18.497+0000 7fdc3eb4e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdc380f6510 0x7fdc381067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:18.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:18.497+0000 7fdc3eb4e700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc38106cd0 con 0x7fdc380f6510 2024-02-19T22:10:18.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:18.498+0000 7fdc3c8ea700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdc380f6510 0x7fdc381067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:18.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:18.498+0000 7fdc3c8ea700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdc380f6510 0x7fdc381067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:52884/0 (socket says 172.21.15.88:52884) 2024-02-19T22:10:18.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:18.498+0000 7fdc3c8ea700 1 -- 172.21.15.88:0/4128493743 learned_addr learned my addr 172.21.15.88:0/4128493743 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:10:18.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:18.498+0000 7fdc3c8ea700 1 -- 172.21.15.88:0/4128493743 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc2c005040 con 0x7fdc380f6510 2024-02-19T22:10:18.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:18.498+0000 7fdc3c8ea700 1 --2- 172.21.15.88:0/4128493743 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdc380f6510 0x7fdc381067e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fdc2c005010 tx=0x7fdc2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:18.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:18.498+0000 7fdc357fa700 1 -- 172.21.15.88:0/4128493743 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc2c009320 con 0x7fdc380f6510 2024-02-19T22:10:18.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:18.498+0000 7fdc3eb4e700 1 -- 172.21.15.88:0/4128493743 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc38106ed0 con 0x7fdc380f6510 2024-02-19T22:10:18.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:18.498+0000 7fdc357fa700 1 -- 172.21.15.88:0/4128493743 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdc2c013380 con 0x7fdc380f6510 2024-02-19T22:10:18.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:18.498+0000 7fdc357fa700 1 -- 172.21.15.88:0/4128493743 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc2c014e50 con 0x7fdc380f6510 2024-02-19T22:10:18.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:18.498+0000 7fdc3eb4e700 1 -- 172.21.15.88:0/4128493743 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc38107360 con 0x7fdc380f6510 2024-02-19T22:10:18.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:18.499+0000 7fdc357fa700 1 -- 172.21.15.88:0/4128493743 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdc2c0095f0 con 0x7fdc380f6510 2024-02-19T22:10:18.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:18.499+0000 7fdc3eb4e700 1 -- 172.21.15.88:0/4128493743 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc38045ad0 con 0x7fdc380f6510 2024-02-19T22:10:18.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:18.499+0000 7fdc357fa700 1 --2- 172.21.15.88:0/4128493743 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdc2802ff10 0x7fdc280323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:18.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:18.499+0000 7fdc357fa700 1 -- 172.21.15.88:0/4128493743 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdc2c040f80 con 0x7fdc380f6510 2024-02-19T22:10:18.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:18.500+0000 7fdc37fff700 1 --2- 172.21.15.88:0/4128493743 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdc2802ff10 0x7fdc280323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:18.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:18.500+0000 7fdc37fff700 1 --2- 172.21.15.88:0/4128493743 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdc2802ff10 0x7fdc280323d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fdc240096a0 tx=0x7fdc24006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:18.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:18.504+0000 7fdc357fa700 1 -- 172.21.15.88:0/4128493743 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdc2c014420 con 0x7fdc380f6510 2024-02-19T22:10:18.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:18.676+0000 7fdc3eb4e700 1 -- 172.21.15.88:0/4128493743 --> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi088", "target": ["mon-mgr", ""]}) v1 -- 0x7fdc380f2e20 con 0x7fdc2802ff10 2024-02-19T22:10:19.300 INFO:teuthology.orchestra.run.smithi088.stdout:Added host 'smithi088' 2024-02-19T22:10:19.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:19.298+0000 7fdc357fa700 1 -- 172.21.15.88:0/4128493743 <== mgr.14162 v2:172.21.15.52:6800/539827537 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7fdc380f2e20 con 0x7fdc2802ff10 2024-02-19T22:10:19.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:19.301+0000 7fdc3eb4e700 1 -- 172.21.15.88:0/4128493743 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdc2802ff10 msgr2=0x7fdc280323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:19.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:19.301+0000 7fdc3eb4e700 1 --2- 172.21.15.88:0/4128493743 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdc2802ff10 0x7fdc280323d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fdc240096a0 tx=0x7fdc24006b40).stop 2024-02-19T22:10:19.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:19.301+0000 7fdc3eb4e700 1 -- 172.21.15.88:0/4128493743 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdc380f6510 msgr2=0x7fdc381067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:19.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:19.301+0000 7fdc3eb4e700 1 --2- 172.21.15.88:0/4128493743 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdc380f6510 0x7fdc381067e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fdc2c005010 tx=0x7fdc2c00b040).stop 2024-02-19T22:10:19.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:19.301+0000 7fdc3eb4e700 1 -- 172.21.15.88:0/4128493743 shutdown_connections 2024-02-19T22:10:19.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:19.301+0000 7fdc3eb4e700 1 --2- 172.21.15.88:0/4128493743 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdc2802ff10 0x7fdc280323d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:19.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:19.301+0000 7fdc3eb4e700 1 --2- 172.21.15.88:0/4128493743 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdc380f6510 0x7fdc381067e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:19.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:19.301+0000 7fdc3eb4e700 1 -- 172.21.15.88:0/4128493743 >> 172.21.15.88:0/4128493743 conn(0x7fdc380f1c70 msgr2=0x7fdc380f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:19.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:19.301+0000 7fdc3eb4e700 1 -- 172.21.15.88:0/4128493743 shutdown_connections 2024-02-19T22:10:19.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:19.301+0000 7fdc3eb4e700 1 -- 172.21.15.88:0/4128493743 wait complete. 2024-02-19T22:10:19.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:10:19 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:10:17.614755+0000 mgr.smithi052.ghqrtk (mgr.14162) 13 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:19.968 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph orch host ls --format=json 2024-02-19T22:10:20.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:10:20 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:10:18.678214+0000 mgr.smithi052.ghqrtk (mgr.14162) 14 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi088", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:10:20.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:10:20 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:10:19.299004+0000 mon.smithi052 (mon.0) 116 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' 2024-02-19T22:10:20.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:10:20 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cephadm 2024-02-19T22:10:19.299450+0000 mgr.smithi052.ghqrtk (mgr.14162) 15 : cephadm [INF] Added host smithi088 2024-02-19T22:10:21.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:10:21 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:10:19.615277+0000 mgr.smithi052.ghqrtk (mgr.14162) 16 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:21.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:21.727+0000 7ff762719700 1 -- 172.21.15.88:0/1559860571 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff75c0f6510 msgr2=0x7ff75c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:21.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:21.727+0000 7ff762719700 1 --2- 172.21.15.88:0/1559860571 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff75c0f6510 0x7ff75c0f6930 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7ff74c009fb0 tx=0x7ff74c009b30).stop 2024-02-19T22:10:21.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:21.727+0000 7ff762719700 1 -- 172.21.15.88:0/1559860571 shutdown_connections 2024-02-19T22:10:21.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:21.727+0000 7ff762719700 1 --2- 172.21.15.88:0/1559860571 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff75c0f6510 0x7ff75c0f6930 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:21.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:21.727+0000 7ff762719700 1 -- 172.21.15.88:0/1559860571 >> 172.21.15.88:0/1559860571 conn(0x7ff75c0f1c70 msgr2=0x7ff75c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:21.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:21.727+0000 7ff762719700 1 -- 172.21.15.88:0/1559860571 shutdown_connections 2024-02-19T22:10:21.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:21.727+0000 7ff762719700 1 -- 172.21.15.88:0/1559860571 wait complete. 2024-02-19T22:10:21.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:21.727+0000 7ff762719700 1 Processor -- start 2024-02-19T22:10:21.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:21.728+0000 7ff762719700 1 -- start start 2024-02-19T22:10:21.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:21.728+0000 7ff762719700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff75c0f6510 0x7ff75c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:21.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:21.728+0000 7ff762719700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff75c069e20 con 0x7ff75c0f6510 2024-02-19T22:10:21.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:21.728+0000 7ff75bfff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff75c0f6510 0x7ff75c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:21.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:21.728+0000 7ff75bfff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff75c0f6510 0x7ff75c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:52902/0 (socket says 172.21.15.88:52902) 2024-02-19T22:10:21.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:21.728+0000 7ff75bfff700 1 -- 172.21.15.88:0/3901365659 learned_addr learned my addr 172.21.15.88:0/3901365659 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:10:21.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:21.729+0000 7ff75bfff700 1 -- 172.21.15.88:0/3901365659 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff74c005040 con 0x7ff75c0f6510 2024-02-19T22:10:21.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:21.729+0000 7ff75bfff700 1 --2- 172.21.15.88:0/3901365659 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff75c0f6510 0x7ff75c06b9a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7ff74c005010 tx=0x7ff74c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:21.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:21.729+0000 7ff758ff9700 1 -- 172.21.15.88:0/3901365659 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff74c009320 con 0x7ff75c0f6510 2024-02-19T22:10:21.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:21.729+0000 7ff762719700 1 -- 172.21.15.88:0/3901365659 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff75c069fc0 con 0x7ff75c0f6510 2024-02-19T22:10:21.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:21.729+0000 7ff758ff9700 1 -- 172.21.15.88:0/3901365659 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff74c013380 con 0x7ff75c0f6510 2024-02-19T22:10:21.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:21.729+0000 7ff758ff9700 1 -- 172.21.15.88:0/3901365659 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff74c014e50 con 0x7ff75c0f6510 2024-02-19T22:10:21.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:21.729+0000 7ff762719700 1 -- 172.21.15.88:0/3901365659 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff75c06a450 con 0x7ff75c0f6510 2024-02-19T22:10:21.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:21.730+0000 7ff758ff9700 1 -- 172.21.15.88:0/3901365659 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff74c0263a0 con 0x7ff75c0f6510 2024-02-19T22:10:21.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:21.730+0000 7ff758ff9700 1 --2- 172.21.15.88:0/3901365659 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7ff74802ff10 0x7ff7480323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:21.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:21.730+0000 7ff758ff9700 1 -- 172.21.15.88:0/3901365659 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff74c040fa0 con 0x7ff75c0f6510 2024-02-19T22:10:21.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:21.730+0000 7ff762719700 1 -- 172.21.15.88:0/3901365659 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff75c059100 con 0x7ff75c0f6510 2024-02-19T22:10:21.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:21.730+0000 7ff75b7fe700 1 --2- 172.21.15.88:0/3901365659 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7ff74802ff10 0x7ff7480323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:21.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:21.731+0000 7ff75b7fe700 1 --2- 172.21.15.88:0/3901365659 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7ff74802ff10 0x7ff7480323d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7ff7440096a0 tx=0x7ff744006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:21.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:21.735+0000 7ff758ff9700 1 -- 172.21.15.88:0/3901365659 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff74c012920 con 0x7ff75c0f6510 2024-02-19T22:10:21.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:21.906+0000 7ff762719700 1 -- 172.21.15.88:0/3901365659 --> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff75c06abc0 con 0x7ff74802ff10 2024-02-19T22:10:21.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:21.908+0000 7ff758ff9700 1 -- 172.21.15.88:0/3901365659 <== mgr.14162 v2:172.21.15.52:6800/539827537 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+156 (secure 0 0 0) 0x7ff75c06abc0 con 0x7ff74802ff10 2024-02-19T22:10:21.909 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:10:21.909 INFO:teuthology.orchestra.run.smithi088.stdout:[{"addr": "172.21.15.52", "hostname": "smithi052", "labels": [], "status": ""}, {"addr": "smithi088", "hostname": "smithi088", "labels": [], "status": ""}] 2024-02-19T22:10:21.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:21.909+0000 7ff762719700 1 -- 172.21.15.88:0/3901365659 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7ff74802ff10 msgr2=0x7ff7480323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:21.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:21.909+0000 7ff762719700 1 --2- 172.21.15.88:0/3901365659 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7ff74802ff10 0x7ff7480323d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7ff7440096a0 tx=0x7ff744006b40).stop 2024-02-19T22:10:21.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:21.909+0000 7ff762719700 1 -- 172.21.15.88:0/3901365659 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff75c0f6510 msgr2=0x7ff75c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:21.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:21.909+0000 7ff762719700 1 --2- 172.21.15.88:0/3901365659 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff75c0f6510 0x7ff75c06b9a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7ff74c005010 tx=0x7ff74c00b040).stop 2024-02-19T22:10:21.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:21.909+0000 7ff762719700 1 -- 172.21.15.88:0/3901365659 shutdown_connections 2024-02-19T22:10:21.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:21.909+0000 7ff762719700 1 --2- 172.21.15.88:0/3901365659 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7ff74802ff10 0x7ff7480323d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:21.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:21.909+0000 7ff762719700 1 --2- 172.21.15.88:0/3901365659 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff75c0f6510 0x7ff75c06b9a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:21.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:21.909+0000 7ff762719700 1 -- 172.21.15.88:0/3901365659 >> 172.21.15.88:0/3901365659 conn(0x7ff75c0f1c70 msgr2=0x7ff75c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:21.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:21.909+0000 7ff762719700 1 -- 172.21.15.88:0/3901365659 shutdown_connections 2024-02-19T22:10:21.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:21.909+0000 7ff762719700 1 -- 172.21.15.88:0/3901365659 wait complete. 2024-02-19T22:10:22.613 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-19T22:10:22.613 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph osd crush tunables default 2024-02-19T22:10:23.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:10:23 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:10:21.615719+0000 mgr.smithi052.ghqrtk (mgr.14162) 17 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:23.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:10:23 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:10:21.908301+0000 mgr.smithi052.ghqrtk (mgr.14162) 18 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T22:10:24.519 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:10:24.503+0000 7f222f641700 1 -- 172.21.15.52:0/2944492105 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f22280f5410 msgr2=0x7f22280f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:24.519 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:10:24.503+0000 7f222f641700 1 --2- 172.21.15.52:0/2944492105 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f22280f5410 0x7f22280f5830 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f2224009fb0 tx=0x7f2224009b30).stop 2024-02-19T22:10:24.519 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:10:24.503+0000 7f222f641700 1 -- 172.21.15.52:0/2944492105 shutdown_connections 2024-02-19T22:10:24.519 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:10:24.503+0000 7f222f641700 1 --2- 172.21.15.52:0/2944492105 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f22280f5410 0x7f22280f5830 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:24.519 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:10:24.503+0000 7f222f641700 1 -- 172.21.15.52:0/2944492105 >> 172.21.15.52:0/2944492105 conn(0x7f22280f0b70 msgr2=0x7f22280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:24.519 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:10:24.503+0000 7f222f641700 1 -- 172.21.15.52:0/2944492105 shutdown_connections 2024-02-19T22:10:24.519 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:10:24.503+0000 7f222f641700 1 -- 172.21.15.52:0/2944492105 wait complete. 2024-02-19T22:10:24.519 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:10:24.504+0000 7f222f641700 1 Processor -- start 2024-02-19T22:10:24.519 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:10:24.504+0000 7f222f641700 1 -- start start 2024-02-19T22:10:24.519 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:10:24.504+0000 7f222f641700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f22280f5410 0x7f22281056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:24.519 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:10:24.504+0000 7f222f641700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2228105bd0 con 0x7f22280f5410 2024-02-19T22:10:24.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:10:24.504+0000 7f222d3dd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f22280f5410 0x7f22281056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:24.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:10:24.504+0000 7f222d3dd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f22280f5410 0x7f22281056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:49450/0 (socket says 172.21.15.52:49450) 2024-02-19T22:10:24.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:10:24.504+0000 7f222d3dd700 1 -- 172.21.15.52:0/2455010591 learned_addr learned my addr 172.21.15.52:0/2455010591 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-02-19T22:10:24.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:10:24.504+0000 7f222d3dd700 1 -- 172.21.15.52:0/2455010591 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2224005040 con 0x7f22280f5410 2024-02-19T22:10:24.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:10:24.504+0000 7f222d3dd700 1 --2- 172.21.15.52:0/2455010591 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f22280f5410 0x7f22281056e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f22240089f0 tx=0x7f222400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:24.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:10:24.505+0000 7f2219ffb700 1 -- 172.21.15.52:0/2455010591 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2224009320 con 0x7f22280f5410 2024-02-19T22:10:24.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:10:24.505+0000 7f222f641700 1 -- 172.21.15.52:0/2455010591 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2228105dd0 con 0x7f22280f5410 2024-02-19T22:10:24.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:10:24.505+0000 7f2219ffb700 1 -- 172.21.15.52:0/2455010591 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2224013380 con 0x7f22280f5410 2024-02-19T22:10:24.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:10:24.505+0000 7f2219ffb700 1 -- 172.21.15.52:0/2455010591 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f222401ed80 con 0x7f22280f5410 2024-02-19T22:10:24.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:10:24.505+0000 7f222f641700 1 -- 172.21.15.52:0/2455010591 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2228106260 con 0x7f22280f5410 2024-02-19T22:10:24.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:10:24.506+0000 7f2219ffb700 1 -- 172.21.15.52:0/2455010591 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2224014420 con 0x7f22280f5410 2024-02-19T22:10:24.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:10:24.506+0000 7f2219ffb700 1 --2- 172.21.15.52:0/2455010591 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f221402ff60 0x7f2214032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:24.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:10:24.506+0000 7f2219ffb700 1 -- 172.21.15.52:0/2455010591 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2224040e10 con 0x7f22280f5410 2024-02-19T22:10:24.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:10:24.506+0000 7f222f641700 1 -- 172.21.15.52:0/2455010591 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2228045ad0 con 0x7f22280f5410 2024-02-19T22:10:24.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:10:24.506+0000 7f222cbdc700 1 --2- 172.21.15.52:0/2455010591 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f221402ff60 0x7f2214032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:24.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:10:24.506+0000 7f222cbdc700 1 --2- 172.21.15.52:0/2455010591 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f221402ff60 0x7f2214032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f221c0096a0 tx=0x7f221c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:24.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:10:24.510+0000 7f2219ffb700 1 -- 172.21.15.52:0/2455010591 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f22240672b0 con 0x7f22280f5410 2024-02-19T22:10:24.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:10:24.673+0000 7f222f641700 1 -- 172.21.15.52:0/2455010591 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f2228106f80 con 0x7f22280f5410 2024-02-19T22:10:25.305 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:10:25.304+0000 7f2219ffb700 1 -- 172.21.15.52:0/2455010591 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7f222401c910 con 0x7f22280f5410 2024-02-19T22:10:25.308 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:10:25.306+0000 7f222f641700 1 -- 172.21.15.52:0/2455010591 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f221402ff60 msgr2=0x7f2214032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:25.308 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:10:25.306+0000 7f222f641700 1 --2- 172.21.15.52:0/2455010591 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f221402ff60 0x7f2214032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f221c0096a0 tx=0x7f221c006b40).stop 2024-02-19T22:10:25.308 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:10:25.306+0000 7f222f641700 1 -- 172.21.15.52:0/2455010591 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f22280f5410 msgr2=0x7f22281056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:25.308 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:10:25.306+0000 7f222f641700 1 --2- 172.21.15.52:0/2455010591 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f22280f5410 0x7f22281056e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f22240089f0 tx=0x7f222400b040).stop 2024-02-19T22:10:25.308 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:10:25.306+0000 7f222f641700 1 -- 172.21.15.52:0/2455010591 shutdown_connections 2024-02-19T22:10:25.308 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:10:25.306+0000 7f222f641700 1 --2- 172.21.15.52:0/2455010591 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f221402ff60 0x7f2214032420 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:25.308 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:10:25.306+0000 7f222f641700 1 --2- 172.21.15.52:0/2455010591 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f22280f5410 0x7f22281056e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:25.308 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:10:25.307+0000 7f222f641700 1 -- 172.21.15.52:0/2455010591 >> 172.21.15.52:0/2455010591 conn(0x7f22280f0b70 msgr2=0x7f22280f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:25.309 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:10:25.307+0000 7f222f641700 1 -- 172.21.15.52:0/2455010591 shutdown_connections 2024-02-19T22:10:25.309 INFO:teuthology.orchestra.run.smithi052.stderr:2024-02-19T22:10:25.307+0000 7f222f641700 1 -- 172.21.15.52:0/2455010591 wait complete. 2024-02-19T22:10:25.309 INFO:teuthology.orchestra.run.smithi052.stderr:adjusted tunables profile to default 2024-02-19T22:10:25.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:10:25 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:10:23.616086+0000 mgr.smithi052.ghqrtk (mgr.14162) 19 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:25.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:10:25 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:10:24.675445+0000 mon.smithi052 (mon.0) 117 : audit [INF] from='client.? 172.21.15.52:0/2455010591' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-19T22:10:26.107 INFO:tasks.cephadm:Adding mon.smithi052 on smithi052 2024-02-19T22:10:26.107 INFO:tasks.cephadm:Adding mon.smithi088 on smithi088 2024-02-19T22:10:26.107 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph orch apply mon '2;smithi052:172.21.15.52=smithi052;smithi088:172.21.15.88=smithi088' 2024-02-19T22:10:26.342 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:10:26 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:10:25.305320+0000 mon.smithi052 (mon.0) 118 : audit [INF] from='client.? 172.21.15.52:0/2455010591' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-19T22:10:26.342 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:10:26 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:10:25.305367+0000 mon.smithi052 (mon.0) 119 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-02-19T22:10:27.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:10:27 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:10:25.616504+0000 mgr.smithi052.ghqrtk (mgr.14162) 20 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:27.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:27.829+0000 7fe7bdc1b700 1 -- 172.21.15.88:0/477861875 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe7b80f5410 msgr2=0x7fe7b80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:27.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:27.829+0000 7fe7bdc1b700 1 --2- 172.21.15.88:0/477861875 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe7b80f5410 0x7fe7b80f5830 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fe7ac009fb0 tx=0x7fe7ac009b30).stop 2024-02-19T22:10:27.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:27.829+0000 7fe7bdc1b700 1 -- 172.21.15.88:0/477861875 shutdown_connections 2024-02-19T22:10:27.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:27.829+0000 7fe7bdc1b700 1 --2- 172.21.15.88:0/477861875 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe7b80f5410 0x7fe7b80f5830 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:27.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:27.829+0000 7fe7bdc1b700 1 -- 172.21.15.88:0/477861875 >> 172.21.15.88:0/477861875 conn(0x7fe7b80f0b70 msgr2=0x7fe7b80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:27.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:27.829+0000 7fe7bdc1b700 1 -- 172.21.15.88:0/477861875 shutdown_connections 2024-02-19T22:10:27.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:27.829+0000 7fe7bdc1b700 1 -- 172.21.15.88:0/477861875 wait complete. 2024-02-19T22:10:27.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:27.830+0000 7fe7bdc1b700 1 Processor -- start 2024-02-19T22:10:27.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:27.830+0000 7fe7bdc1b700 1 -- start start 2024-02-19T22:10:27.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:27.830+0000 7fe7bdc1b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe7b80f5410 0x7fe7b81067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:27.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:27.830+0000 7fe7bdc1b700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b8106cc0 con 0x7fe7b80f5410 2024-02-19T22:10:27.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:27.830+0000 7fe7b77fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe7b80f5410 0x7fe7b81067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:27.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:27.830+0000 7fe7b77fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe7b80f5410 0x7fe7b81067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:51174/0 (socket says 172.21.15.88:51174) 2024-02-19T22:10:27.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:27.830+0000 7fe7b77fe700 1 -- 172.21.15.88:0/3104452148 learned_addr learned my addr 172.21.15.88:0/3104452148 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:10:27.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:27.830+0000 7fe7b77fe700 1 -- 172.21.15.88:0/3104452148 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7ac005040 con 0x7fe7b80f5410 2024-02-19T22:10:27.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:27.831+0000 7fe7b77fe700 1 --2- 172.21.15.88:0/3104452148 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe7b80f5410 0x7fe7b81067d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fe7ac005010 tx=0x7fe7ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:27.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:27.831+0000 7fe79ffff700 1 -- 172.21.15.88:0/3104452148 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7ac009320 con 0x7fe7b80f5410 2024-02-19T22:10:27.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:27.831+0000 7fe7bdc1b700 1 -- 172.21.15.88:0/3104452148 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7b8106ec0 con 0x7fe7b80f5410 2024-02-19T22:10:27.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:27.831+0000 7fe79ffff700 1 -- 172.21.15.88:0/3104452148 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe7ac013380 con 0x7fe7b80f5410 2024-02-19T22:10:27.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:27.831+0000 7fe79ffff700 1 -- 172.21.15.88:0/3104452148 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7ac014e50 con 0x7fe7b80f5410 2024-02-19T22:10:27.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:27.831+0000 7fe7bdc1b700 1 -- 172.21.15.88:0/3104452148 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7b8107350 con 0x7fe7b80f5410 2024-02-19T22:10:27.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:27.832+0000 7fe79ffff700 1 -- 172.21.15.88:0/3104452148 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe7ac0263a0 con 0x7fe7b80f5410 2024-02-19T22:10:27.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:27.832+0000 7fe7bdc1b700 1 -- 172.21.15.88:0/3104452148 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7b8045ad0 con 0x7fe7b80f5410 2024-02-19T22:10:27.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:27.832+0000 7fe79ffff700 1 --2- 172.21.15.88:0/3104452148 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fe7a002ff10 0x7fe7a00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:27.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:27.832+0000 7fe79ffff700 1 -- 172.21.15.88:0/3104452148 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe7ac041260 con 0x7fe7b80f5410 2024-02-19T22:10:27.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:27.832+0000 7fe7b6ffd700 1 --2- 172.21.15.88:0/3104452148 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fe7a002ff10 0x7fe7a00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:27.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:27.832+0000 7fe7b6ffd700 1 --2- 172.21.15.88:0/3104452148 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fe7a002ff10 0x7fe7a00323d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fe7a80096a0 tx=0x7fe7a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:27.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:27.836+0000 7fe79ffff700 1 -- 172.21.15.88:0/3104452148 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe7ac01ccf0 con 0x7fe7b80f5410 2024-02-19T22:10:28.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:28.009+0000 7fe7bdc1b700 1 -- 172.21.15.88:0/3104452148 --> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi052:172.21.15.52=smithi052;smithi088:172.21.15.88=smithi088", "target": ["mon-mgr", ""]}) v1 -- 0x7fe7b80f2a00 con 0x7fe7a002ff10 2024-02-19T22:10:28.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:28.015+0000 7fe79ffff700 1 -- 172.21.15.88:0/3104452148 <== mgr.14162 v2:172.21.15.52:6800/539827537 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fe7b80f2a00 con 0x7fe7a002ff10 2024-02-19T22:10:28.015 INFO:teuthology.orchestra.run.smithi088.stdout:Scheduled mon update... 2024-02-19T22:10:28.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:28.016+0000 7fe7bdc1b700 1 -- 172.21.15.88:0/3104452148 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fe7a002ff10 msgr2=0x7fe7a00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:28.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:28.016+0000 7fe7bdc1b700 1 --2- 172.21.15.88:0/3104452148 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fe7a002ff10 0x7fe7a00323d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fe7a80096a0 tx=0x7fe7a8006b40).stop 2024-02-19T22:10:28.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:28.016+0000 7fe7bdc1b700 1 -- 172.21.15.88:0/3104452148 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe7b80f5410 msgr2=0x7fe7b81067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:28.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:28.016+0000 7fe7bdc1b700 1 --2- 172.21.15.88:0/3104452148 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe7b80f5410 0x7fe7b81067d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fe7ac005010 tx=0x7fe7ac00b040).stop 2024-02-19T22:10:28.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:28.016+0000 7fe7bdc1b700 1 -- 172.21.15.88:0/3104452148 shutdown_connections 2024-02-19T22:10:28.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:28.016+0000 7fe7bdc1b700 1 --2- 172.21.15.88:0/3104452148 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fe7a002ff10 0x7fe7a00323d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:28.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:28.016+0000 7fe7bdc1b700 1 --2- 172.21.15.88:0/3104452148 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe7b80f5410 0x7fe7b81067d0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:28.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:28.016+0000 7fe7bdc1b700 1 -- 172.21.15.88:0/3104452148 >> 172.21.15.88:0/3104452148 conn(0x7fe7b80f0b70 msgr2=0x7fe7b805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:28.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:28.016+0000 7fe7bdc1b700 1 -- 172.21.15.88:0/3104452148 shutdown_connections 2024-02-19T22:10:28.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:28.016+0000 7fe7bdc1b700 1 -- 172.21.15.88:0/3104452148 wait complete. 2024-02-19T22:10:29.118 DEBUG:teuthology.orchestra.run.smithi088:mon.smithi088> sudo journalctl -f -n 0 -u ceph-56739a54-cf73-11ee-95bb-87774f69a715@mon.smithi088.service 2024-02-19T22:10:29.120 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:10:29.120 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:10:29.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:-- Logs begin at Mon 2024-02-19 21:54:54 UTC. -- 2024-02-19T22:10:29.342 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:10:29 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:10:27.616804+0000 mgr.smithi052.ghqrtk (mgr.14162) 21 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:29.342 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:10:29 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:10:28.011175+0000 mgr.smithi052.ghqrtk (mgr.14162) 22 : audit [DBG] from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi052:172.21.15.52=smithi052;smithi088:172.21.15.88=smithi088", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:10:29.342 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:10:29 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cephadm 2024-02-19T22:10:28.012667+0000 mgr.smithi052.ghqrtk (mgr.14162) 23 : cephadm [INF] Saving service mon spec with placement smithi052:172.21.15.52=smithi052;smithi088:172.21.15.88=smithi088;count:2 2024-02-19T22:10:29.343 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:10:29 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:10:28.015183+0000 mon.smithi052 (mon.0) 120 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' 2024-02-19T22:10:30.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:30.861+0000 7f5a5318e700 1 -- 172.21.15.88:0/2834045902 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5a4c0f5410 msgr2=0x7f5a4c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:30.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:30.861+0000 7f5a5318e700 1 --2- 172.21.15.88:0/2834045902 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5a4c0f5410 0x7f5a4c0f5830 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f5a40009fb0 tx=0x7f5a40009b30).stop 2024-02-19T22:10:30.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:30.861+0000 7f5a5318e700 1 -- 172.21.15.88:0/2834045902 shutdown_connections 2024-02-19T22:10:30.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:30.861+0000 7f5a5318e700 1 --2- 172.21.15.88:0/2834045902 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5a4c0f5410 0x7f5a4c0f5830 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:30.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:30.861+0000 7f5a5318e700 1 -- 172.21.15.88:0/2834045902 >> 172.21.15.88:0/2834045902 conn(0x7f5a4c0f0b70 msgr2=0x7f5a4c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:30.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:30.861+0000 7f5a5318e700 1 -- 172.21.15.88:0/2834045902 shutdown_connections 2024-02-19T22:10:30.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:30.861+0000 7f5a5318e700 1 -- 172.21.15.88:0/2834045902 wait complete. 2024-02-19T22:10:30.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:30.861+0000 7f5a5318e700 1 Processor -- start 2024-02-19T22:10:30.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:30.861+0000 7f5a5318e700 1 -- start start 2024-02-19T22:10:30.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:30.862+0000 7f5a5318e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5a4c0f5410 0x7f5a4c0fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:30.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:30.862+0000 7f5a5318e700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4c0fb960 con 0x7f5a4c0f5410 2024-02-19T22:10:30.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:30.862+0000 7f5a50f2a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5a4c0f5410 0x7f5a4c0fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:30.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:30.862+0000 7f5a50f2a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5a4c0f5410 0x7f5a4c0fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:51196/0 (socket says 172.21.15.88:51196) 2024-02-19T22:10:30.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:30.862+0000 7f5a50f2a700 1 -- 172.21.15.88:0/3402827471 learned_addr learned my addr 172.21.15.88:0/3402827471 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:10:30.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:30.862+0000 7f5a50f2a700 1 -- 172.21.15.88:0/3402827471 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a40005040 con 0x7f5a4c0f5410 2024-02-19T22:10:30.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:30.862+0000 7f5a50f2a700 1 --2- 172.21.15.88:0/3402827471 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5a4c0f5410 0x7f5a4c0fd4e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f5a40005010 tx=0x7f5a4000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:30.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:30.862+0000 7f5a497fa700 1 -- 172.21.15.88:0/3402827471 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a40009320 con 0x7f5a4c0f5410 2024-02-19T22:10:30.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:30.863+0000 7f5a5318e700 1 -- 172.21.15.88:0/3402827471 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a4c0fbb00 con 0x7f5a4c0f5410 2024-02-19T22:10:30.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:30.863+0000 7f5a497fa700 1 -- 172.21.15.88:0/3402827471 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5a40013380 con 0x7f5a4c0f5410 2024-02-19T22:10:30.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:30.863+0000 7f5a497fa700 1 -- 172.21.15.88:0/3402827471 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a40014e50 con 0x7f5a4c0f5410 2024-02-19T22:10:30.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:30.863+0000 7f5a5318e700 1 -- 172.21.15.88:0/3402827471 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a4c0fbf90 con 0x7f5a4c0f5410 2024-02-19T22:10:30.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:30.863+0000 7f5a497fa700 1 -- 172.21.15.88:0/3402827471 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5a400095f0 con 0x7f5a4c0f5410 2024-02-19T22:10:30.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:30.863+0000 7f5a5318e700 1 -- 172.21.15.88:0/3402827471 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a4c059100 con 0x7f5a4c0f5410 2024-02-19T22:10:30.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:30.863+0000 7f5a497fa700 1 --2- 172.21.15.88:0/3402827471 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f5a3c02ff10 0x7f5a3c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:30.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:30.864+0000 7f5a497fa700 1 -- 172.21.15.88:0/3402827471 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5a40040e90 con 0x7f5a4c0f5410 2024-02-19T22:10:30.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:30.864+0000 7f5a4bfff700 1 --2- 172.21.15.88:0/3402827471 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f5a3c02ff10 0x7f5a3c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:30.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:30.864+0000 7f5a4bfff700 1 --2- 172.21.15.88:0/3402827471 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f5a3c02ff10 0x7f5a3c0323d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f5a380096a0 tx=0x7f5a38006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:30.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:30.868+0000 7f5a497fa700 1 -- 172.21.15.88:0/3402827471 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5a40014420 con 0x7f5a4c0f5410 2024-02-19T22:10:31.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:31.071+0000 7f5a5318e700 1 -- 172.21.15.88:0/3402827471 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5a4c045ad0 con 0x7f5a4c0f5410 2024-02-19T22:10:31.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:31.071+0000 7f5a497fa700 1 -- 172.21.15.88:0/3402827471 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5a40010230 con 0x7f5a4c0f5410 2024-02-19T22:10:31.073 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:10:31.073 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:10:31.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:31.073+0000 7f5a5318e700 1 -- 172.21.15.88:0/3402827471 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f5a3c02ff10 msgr2=0x7f5a3c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:31.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:31.073+0000 7f5a5318e700 1 --2- 172.21.15.88:0/3402827471 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f5a3c02ff10 0x7f5a3c0323d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f5a380096a0 tx=0x7f5a38006b40).stop 2024-02-19T22:10:31.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:31.073+0000 7f5a5318e700 1 -- 172.21.15.88:0/3402827471 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5a4c0f5410 msgr2=0x7f5a4c0fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:31.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:31.073+0000 7f5a5318e700 1 --2- 172.21.15.88:0/3402827471 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5a4c0f5410 0x7f5a4c0fd4e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f5a40005010 tx=0x7f5a4000b040).stop 2024-02-19T22:10:31.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:31.073+0000 7f5a5318e700 1 -- 172.21.15.88:0/3402827471 shutdown_connections 2024-02-19T22:10:31.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:31.073+0000 7f5a5318e700 1 --2- 172.21.15.88:0/3402827471 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f5a3c02ff10 0x7f5a3c0323d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:31.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:31.073+0000 7f5a5318e700 1 --2- 172.21.15.88:0/3402827471 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5a4c0f5410 0x7f5a4c0fd4e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:31.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:31.073+0000 7f5a5318e700 1 -- 172.21.15.88:0/3402827471 >> 172.21.15.88:0/3402827471 conn(0x7f5a4c0f0b70 msgr2=0x7f5a4c05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:31.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:31.073+0000 7f5a5318e700 1 -- 172.21.15.88:0/3402827471 shutdown_connections 2024-02-19T22:10:31.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:31.073+0000 7f5a5318e700 1 -- 172.21.15.88:0/3402827471 wait complete. 2024-02-19T22:10:31.075 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:10:31.342 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:10:31 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:10:29.617046+0000 mgr.smithi052.ghqrtk (mgr.14162) 24 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:31.342 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:10:31 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:10:31.072592+0000 mon.smithi052 (mon.0) 121 : audit [DBG] from='client.? 172.21.15.88:0/3402827471' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:10:32.684 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:10:32.685 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:10:33.342 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:10:33 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:10:31.617465+0000 mgr.smithi052.ghqrtk (mgr.14162) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:34.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:34.374+0000 7f0669c28700 1 -- 172.21.15.88:0/2023526983 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0664060280 msgr2=0x7f06640f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:34.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:34.374+0000 7f0669c28700 1 --2- 172.21.15.88:0/2023526983 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0664060280 0x7f06640f90f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f0658009fb0 tx=0x7f0658009b30).stop 2024-02-19T22:10:34.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:34.374+0000 7f0669c28700 1 -- 172.21.15.88:0/2023526983 shutdown_connections 2024-02-19T22:10:34.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:34.374+0000 7f0669c28700 1 --2- 172.21.15.88:0/2023526983 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0664060280 0x7f06640f90f0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:34.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:34.374+0000 7f0669c28700 1 -- 172.21.15.88:0/2023526983 >> 172.21.15.88:0/2023526983 conn(0x7f06640f0b70 msgr2=0x7f06640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:34.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:34.374+0000 7f0669c28700 1 -- 172.21.15.88:0/2023526983 shutdown_connections 2024-02-19T22:10:34.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:34.374+0000 7f0669c28700 1 -- 172.21.15.88:0/2023526983 wait complete. 2024-02-19T22:10:34.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:34.375+0000 7f0669c28700 1 Processor -- start 2024-02-19T22:10:34.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:34.375+0000 7f0669c28700 1 -- start start 2024-02-19T22:10:34.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:34.375+0000 7f0669c28700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0664060280 0x7f0664106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:34.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:34.375+0000 7f0669c28700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0664106c70 con 0x7f0664060280 2024-02-19T22:10:34.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:34.375+0000 7f06637fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0664060280 0x7f0664106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:34.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:34.375+0000 7f06637fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0664060280 0x7f0664106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:53906/0 (socket says 172.21.15.88:53906) 2024-02-19T22:10:34.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:34.375+0000 7f06637fe700 1 -- 172.21.15.88:0/1344279012 learned_addr learned my addr 172.21.15.88:0/1344279012 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:10:34.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:34.375+0000 7f06637fe700 1 -- 172.21.15.88:0/1344279012 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0658005040 con 0x7f0664060280 2024-02-19T22:10:34.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:34.376+0000 7f06637fe700 1 --2- 172.21.15.88:0/1344279012 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0664060280 0x7f0664106780 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f06580089f0 tx=0x7f065800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:34.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:34.376+0000 7f064bfff700 1 -- 172.21.15.88:0/1344279012 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0658009320 con 0x7f0664060280 2024-02-19T22:10:34.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:34.376+0000 7f0669c28700 1 -- 172.21.15.88:0/1344279012 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0664106e70 con 0x7f0664060280 2024-02-19T22:10:34.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:34.376+0000 7f064bfff700 1 -- 172.21.15.88:0/1344279012 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0658013380 con 0x7f0664060280 2024-02-19T22:10:34.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:34.376+0000 7f064bfff700 1 -- 172.21.15.88:0/1344279012 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0658014e20 con 0x7f0664060280 2024-02-19T22:10:34.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:34.376+0000 7f0669c28700 1 -- 172.21.15.88:0/1344279012 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0664107300 con 0x7f0664060280 2024-02-19T22:10:34.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:34.377+0000 7f064bfff700 1 -- 172.21.15.88:0/1344279012 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0658014420 con 0x7f0664060280 2024-02-19T22:10:34.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:34.377+0000 7f0669c28700 1 -- 172.21.15.88:0/1344279012 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0664045ad0 con 0x7f0664060280 2024-02-19T22:10:34.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:34.377+0000 7f064bfff700 1 --2- 172.21.15.88:0/1344279012 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f064c02ff10 0x7f064c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:34.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:34.377+0000 7f064bfff700 1 -- 172.21.15.88:0/1344279012 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0658040fc0 con 0x7f0664060280 2024-02-19T22:10:34.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:34.377+0000 7f0662ffd700 1 --2- 172.21.15.88:0/1344279012 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f064c02ff10 0x7f064c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:34.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:34.377+0000 7f0662ffd700 1 --2- 172.21.15.88:0/1344279012 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f064c02ff10 0x7f064c0323d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f06540096a0 tx=0x7f0654006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:34.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:34.382+0000 7f064bfff700 1 -- 172.21.15.88:0/1344279012 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f065806d2b0 con 0x7f0664060280 2024-02-19T22:10:34.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:34.587+0000 7f0669c28700 1 -- 172.21.15.88:0/1344279012 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f06641081b0 con 0x7f0664060280 2024-02-19T22:10:34.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:34.587+0000 7f064bfff700 1 -- 172.21.15.88:0/1344279012 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f06580095f0 con 0x7f0664060280 2024-02-19T22:10:34.589 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:10:34.589 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:10:34.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:34.589+0000 7f0669c28700 1 -- 172.21.15.88:0/1344279012 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f064c02ff10 msgr2=0x7f064c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:34.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:34.589+0000 7f0669c28700 1 --2- 172.21.15.88:0/1344279012 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f064c02ff10 0x7f064c0323d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f06540096a0 tx=0x7f0654006b40).stop 2024-02-19T22:10:34.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:34.589+0000 7f0669c28700 1 -- 172.21.15.88:0/1344279012 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0664060280 msgr2=0x7f0664106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:34.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:34.589+0000 7f0669c28700 1 --2- 172.21.15.88:0/1344279012 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0664060280 0x7f0664106780 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f06580089f0 tx=0x7f065800b040).stop 2024-02-19T22:10:34.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:34.589+0000 7f0669c28700 1 -- 172.21.15.88:0/1344279012 shutdown_connections 2024-02-19T22:10:34.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:34.589+0000 7f0669c28700 1 --2- 172.21.15.88:0/1344279012 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f064c02ff10 0x7f064c0323d0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:34.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:34.589+0000 7f0669c28700 1 --2- 172.21.15.88:0/1344279012 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0664060280 0x7f0664106780 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:34.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:34.589+0000 7f0669c28700 1 -- 172.21.15.88:0/1344279012 >> 172.21.15.88:0/1344279012 conn(0x7f06640f0b70 msgr2=0x7f06640f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:34.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:34.589+0000 7f0669c28700 1 -- 172.21.15.88:0/1344279012 shutdown_connections 2024-02-19T22:10:34.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:34.589+0000 7f0669c28700 1 -- 172.21.15.88:0/1344279012 wait complete. 2024-02-19T22:10:34.591 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:10:35.342 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:10:35 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:10:33.617933+0000 mgr.smithi052.ghqrtk (mgr.14162) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:35.342 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:10:35 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:10:34.588726+0000 mon.smithi052 (mon.0) 122 : audit [DBG] from='client.? 172.21.15.88:0/1344279012' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:10:36.249 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:10:36.249 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:10:37.342 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:10:37 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:10:35.618465+0000 mgr.smithi052.ghqrtk (mgr.14162) 27 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:37.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:37.957+0000 7fb408a08700 1 -- 172.21.15.88:0/2225544855 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb4040f51a0 msgr2=0x7fb4040f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:37.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:37.957+0000 7fb408a08700 1 --2- 172.21.15.88:0/2225544855 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb4040f51a0 0x7fb4040f55c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fb3f4009fb0 tx=0x7fb3f4009b30).stop 2024-02-19T22:10:37.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:37.957+0000 7fb408a08700 1 -- 172.21.15.88:0/2225544855 shutdown_connections 2024-02-19T22:10:37.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:37.957+0000 7fb408a08700 1 --2- 172.21.15.88:0/2225544855 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb4040f51a0 0x7fb4040f55c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:37.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:37.957+0000 7fb408a08700 1 -- 172.21.15.88:0/2225544855 >> 172.21.15.88:0/2225544855 conn(0x7fb4040f09a0 msgr2=0x7fb4040f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:37.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:37.957+0000 7fb408a08700 1 -- 172.21.15.88:0/2225544855 shutdown_connections 2024-02-19T22:10:37.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:37.957+0000 7fb408a08700 1 -- 172.21.15.88:0/2225544855 wait complete. 2024-02-19T22:10:37.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:37.957+0000 7fb408a08700 1 Processor -- start 2024-02-19T22:10:37.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:37.957+0000 7fb408a08700 1 -- start start 2024-02-19T22:10:37.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:37.957+0000 7fb408a08700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb4040f51a0 0x7fb4041087d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:37.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:37.957+0000 7fb408a08700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb404108cc0 con 0x7fb4040f51a0 2024-02-19T22:10:37.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:37.958+0000 7fb402d9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb4040f51a0 0x7fb4041087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:37.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:37.958+0000 7fb402d9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb4040f51a0 0x7fb4041087d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:53934/0 (socket says 172.21.15.88:53934) 2024-02-19T22:10:37.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:37.958+0000 7fb402d9d700 1 -- 172.21.15.88:0/3625256422 learned_addr learned my addr 172.21.15.88:0/3625256422 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:10:37.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:37.958+0000 7fb402d9d700 1 -- 172.21.15.88:0/3625256422 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f4005040 con 0x7fb4040f51a0 2024-02-19T22:10:37.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:37.958+0000 7fb402d9d700 1 --2- 172.21.15.88:0/3625256422 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb4040f51a0 0x7fb4041087d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fb3f40089f0 tx=0x7fb3f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:37.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:37.958+0000 7fb3eb7fe700 1 -- 172.21.15.88:0/3625256422 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3f4009320 con 0x7fb4040f51a0 2024-02-19T22:10:37.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:37.958+0000 7fb408a08700 1 -- 172.21.15.88:0/3625256422 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb404108ec0 con 0x7fb4040f51a0 2024-02-19T22:10:37.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:37.958+0000 7fb3eb7fe700 1 -- 172.21.15.88:0/3625256422 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb3f4013380 con 0x7fb4040f51a0 2024-02-19T22:10:37.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:37.958+0000 7fb3eb7fe700 1 -- 172.21.15.88:0/3625256422 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3f4014e20 con 0x7fb4040f51a0 2024-02-19T22:10:37.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:37.958+0000 7fb408a08700 1 -- 172.21.15.88:0/3625256422 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb404109350 con 0x7fb4040f51a0 2024-02-19T22:10:37.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:37.959+0000 7fb3eb7fe700 1 -- 172.21.15.88:0/3625256422 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb3f4014420 con 0x7fb4040f51a0 2024-02-19T22:10:37.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:37.959+0000 7fb408a08700 1 -- 172.21.15.88:0/3625256422 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb404045ad0 con 0x7fb4040f51a0 2024-02-19T22:10:37.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:37.959+0000 7fb3eb7fe700 1 --2- 172.21.15.88:0/3625256422 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fb3ec02ff10 0x7fb3ec0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:37.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:37.959+0000 7fb3eb7fe700 1 -- 172.21.15.88:0/3625256422 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb3f4040ef0 con 0x7fb4040f51a0 2024-02-19T22:10:37.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:37.959+0000 7fb40259c700 1 --2- 172.21.15.88:0/3625256422 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fb3ec02ff10 0x7fb3ec0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:37.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:37.960+0000 7fb40259c700 1 --2- 172.21.15.88:0/3625256422 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fb3ec02ff10 0x7fb3ec0323d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fb3f80096a0 tx=0x7fb3f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:37.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:37.964+0000 7fb3eb7fe700 1 -- 172.21.15.88:0/3625256422 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb3f4014790 con 0x7fb4040f51a0 2024-02-19T22:10:38.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:38.171+0000 7fb408a08700 1 -- 172.21.15.88:0/3625256422 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb40410a1a0 con 0x7fb4040f51a0 2024-02-19T22:10:38.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:38.172+0000 7fb3eb7fe700 1 -- 172.21.15.88:0/3625256422 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb3f401d7f0 con 0x7fb4040f51a0 2024-02-19T22:10:38.173 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:10:38.173 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:10:38.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:38.173+0000 7fb408a08700 1 -- 172.21.15.88:0/3625256422 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fb3ec02ff10 msgr2=0x7fb3ec0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:38.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:38.173+0000 7fb408a08700 1 --2- 172.21.15.88:0/3625256422 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fb3ec02ff10 0x7fb3ec0323d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fb3f80096a0 tx=0x7fb3f8006b40).stop 2024-02-19T22:10:38.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:38.173+0000 7fb408a08700 1 -- 172.21.15.88:0/3625256422 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb4040f51a0 msgr2=0x7fb4041087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:38.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:38.173+0000 7fb408a08700 1 --2- 172.21.15.88:0/3625256422 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb4040f51a0 0x7fb4041087d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fb3f40089f0 tx=0x7fb3f400b040).stop 2024-02-19T22:10:38.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:38.173+0000 7fb408a08700 1 -- 172.21.15.88:0/3625256422 shutdown_connections 2024-02-19T22:10:38.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:38.173+0000 7fb408a08700 1 --2- 172.21.15.88:0/3625256422 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fb3ec02ff10 0x7fb3ec0323d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:38.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:38.173+0000 7fb408a08700 1 --2- 172.21.15.88:0/3625256422 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb4040f51a0 0x7fb4041087d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:38.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:38.173+0000 7fb408a08700 1 -- 172.21.15.88:0/3625256422 >> 172.21.15.88:0/3625256422 conn(0x7fb4040f09a0 msgr2=0x7fb40405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:38.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:38.173+0000 7fb408a08700 1 -- 172.21.15.88:0/3625256422 shutdown_connections 2024-02-19T22:10:38.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:38.173+0000 7fb408a08700 1 -- 172.21.15.88:0/3625256422 wait complete. 2024-02-19T22:10:38.176 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:10:39.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:10:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:10:37.618946+0000 mgr.smithi052.ghqrtk (mgr.14162) 28 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:39.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:10:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:10:38.173262+0000 mon.smithi052 (mon.0) 123 : audit [DBG] from='client.? 172.21.15.88:0/3625256422' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:10:39.713 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:10:39.713 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:10:41.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:10:41 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:10:39.619465+0000 mgr.smithi052.ghqrtk (mgr.14162) 29 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:42.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:42.319+0000 7f7dc8dae700 1 -- 172.21.15.88:0/2845346025 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7dc40f5410 msgr2=0x7f7dc40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:42.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:42.319+0000 7f7dc8dae700 1 --2- 172.21.15.88:0/2845346025 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7dc40f5410 0x7f7dc40f5830 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f7db4009fb0 tx=0x7f7db4009b30).stop 2024-02-19T22:10:42.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:42.320+0000 7f7dc8dae700 1 -- 172.21.15.88:0/2845346025 shutdown_connections 2024-02-19T22:10:42.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:42.320+0000 7f7dc8dae700 1 --2- 172.21.15.88:0/2845346025 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7dc40f5410 0x7f7dc40f5830 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:42.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:42.320+0000 7f7dc8dae700 1 -- 172.21.15.88:0/2845346025 >> 172.21.15.88:0/2845346025 conn(0x7f7dc40f0b70 msgr2=0x7f7dc40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:42.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:42.320+0000 7f7dc8dae700 1 -- 172.21.15.88:0/2845346025 shutdown_connections 2024-02-19T22:10:42.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:42.320+0000 7f7dc8dae700 1 -- 172.21.15.88:0/2845346025 wait complete. 2024-02-19T22:10:42.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:42.320+0000 7f7dc8dae700 1 Processor -- start 2024-02-19T22:10:42.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:42.320+0000 7f7dc8dae700 1 -- start start 2024-02-19T22:10:42.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:42.320+0000 7f7dc8dae700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7dc40f5410 0x7f7dc40fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:42.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:42.320+0000 7f7dc8dae700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc40fb960 con 0x7f7dc40f5410 2024-02-19T22:10:42.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:42.320+0000 7f7dc259c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7dc40f5410 0x7f7dc40fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:42.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:42.320+0000 7f7dc259c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7dc40f5410 0x7f7dc40fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:53954/0 (socket says 172.21.15.88:53954) 2024-02-19T22:10:42.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:42.320+0000 7f7dc259c700 1 -- 172.21.15.88:0/3527031952 learned_addr learned my addr 172.21.15.88:0/3527031952 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:10:42.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:42.321+0000 7f7dc259c700 1 -- 172.21.15.88:0/3527031952 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7db4005040 con 0x7f7dc40f5410 2024-02-19T22:10:42.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:42.321+0000 7f7dc259c700 1 --2- 172.21.15.88:0/3527031952 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7dc40f5410 0x7f7dc40fd4e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f7db4005010 tx=0x7f7db400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:42.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:42.321+0000 7f7dbaffd700 1 -- 172.21.15.88:0/3527031952 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7db4009320 con 0x7f7dc40f5410 2024-02-19T22:10:42.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:42.321+0000 7f7dbaffd700 1 -- 172.21.15.88:0/3527031952 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7db4013380 con 0x7f7dc40f5410 2024-02-19T22:10:42.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:42.321+0000 7f7dbaffd700 1 -- 172.21.15.88:0/3527031952 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7db4014e50 con 0x7f7dc40f5410 2024-02-19T22:10:42.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:42.321+0000 7f7dc8dae700 1 -- 172.21.15.88:0/3527031952 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dc40fbb00 con 0x7f7dc40f5410 2024-02-19T22:10:42.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:42.321+0000 7f7dc8dae700 1 -- 172.21.15.88:0/3527031952 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dc40fbf90 con 0x7f7dc40f5410 2024-02-19T22:10:42.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:42.322+0000 7f7dc8dae700 1 -- 172.21.15.88:0/3527031952 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dc4059100 con 0x7f7dc40f5410 2024-02-19T22:10:42.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:42.322+0000 7f7dbaffd700 1 -- 172.21.15.88:0/3527031952 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7db40095f0 con 0x7f7dc40f5410 2024-02-19T22:10:42.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:42.322+0000 7f7dbaffd700 1 --2- 172.21.15.88:0/3527031952 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f7db002ff10 0x7f7db00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:42.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:42.322+0000 7f7dbaffd700 1 -- 172.21.15.88:0/3527031952 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7db4040f70 con 0x7f7dc40f5410 2024-02-19T22:10:42.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:42.322+0000 7f7dc1d9b700 1 --2- 172.21.15.88:0/3527031952 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f7db002ff10 0x7f7db00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:42.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:42.323+0000 7f7dc1d9b700 1 --2- 172.21.15.88:0/3527031952 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f7db002ff10 0x7f7db00323d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f7dac0096a0 tx=0x7f7dac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:42.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:42.327+0000 7f7dbaffd700 1 -- 172.21.15.88:0/3527031952 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7db4014420 con 0x7f7dc40f5410 2024-02-19T22:10:42.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:42.533+0000 7f7dc8dae700 1 -- 172.21.15.88:0/3527031952 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7dc40fafa0 con 0x7f7dc40f5410 2024-02-19T22:10:42.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:42.533+0000 7f7dbaffd700 1 -- 172.21.15.88:0/3527031952 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7db401d980 con 0x7f7dc40f5410 2024-02-19T22:10:42.535 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:10:42.535 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:10:42.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:42.535+0000 7f7dc8dae700 1 -- 172.21.15.88:0/3527031952 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f7db002ff10 msgr2=0x7f7db00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:42.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:42.535+0000 7f7dc8dae700 1 --2- 172.21.15.88:0/3527031952 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f7db002ff10 0x7f7db00323d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f7dac0096a0 tx=0x7f7dac006b40).stop 2024-02-19T22:10:42.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:42.535+0000 7f7dc8dae700 1 -- 172.21.15.88:0/3527031952 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7dc40f5410 msgr2=0x7f7dc40fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:42.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:42.535+0000 7f7dc8dae700 1 --2- 172.21.15.88:0/3527031952 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7dc40f5410 0x7f7dc40fd4e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f7db4005010 tx=0x7f7db400b040).stop 2024-02-19T22:10:42.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:42.535+0000 7f7dc8dae700 1 -- 172.21.15.88:0/3527031952 shutdown_connections 2024-02-19T22:10:42.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:42.535+0000 7f7dc8dae700 1 --2- 172.21.15.88:0/3527031952 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f7db002ff10 0x7f7db00323d0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:42.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:42.535+0000 7f7dc8dae700 1 --2- 172.21.15.88:0/3527031952 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7dc40f5410 0x7f7dc40fd4e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:42.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:42.535+0000 7f7dc8dae700 1 -- 172.21.15.88:0/3527031952 >> 172.21.15.88:0/3527031952 conn(0x7f7dc40f0b70 msgr2=0x7f7dc40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:42.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:42.535+0000 7f7dc8dae700 1 -- 172.21.15.88:0/3527031952 shutdown_connections 2024-02-19T22:10:42.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:42.535+0000 7f7dc8dae700 1 -- 172.21.15.88:0/3527031952 wait complete. 2024-02-19T22:10:42.537 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:10:43.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:10:43 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:10:41.619812+0000 mgr.smithi052.ghqrtk (mgr.14162) 30 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:43.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:10:43 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:10:42.534827+0000 mon.smithi052 (mon.0) 124 : audit [DBG] from='client.? 172.21.15.88:0/3527031952' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:10:44.077 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:10:44.077 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:10:44.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:10:44 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:10:43.745076+0000 mon.smithi052 (mon.0) 125 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi052.ghqrtk/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:10:44.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:10:44 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:10:43.748698+0000 mon.smithi052 (mon.0) 126 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi052.ghqrtk/trash_purge_schedule"}]: dispatch 2024-02-19T22:10:45.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:10:45 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:10:43.620057+0000 mgr.smithi052.ghqrtk (mgr.14162) 31 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:47.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:10:47 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:10:45.620532+0000 mgr.smithi052.ghqrtk (mgr.14162) 32 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:47.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:47.494+0000 7f6d9bf94700 1 -- 172.21.15.88:0/3697536149 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6d940f5410 msgr2=0x7f6d940f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:47.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:47.494+0000 7f6d9bf94700 1 --2- 172.21.15.88:0/3697536149 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6d940f5410 0x7f6d940f5830 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f6d88009fb0 tx=0x7f6d88009b30).stop 2024-02-19T22:10:47.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:47.494+0000 7f6d9bf94700 1 -- 172.21.15.88:0/3697536149 shutdown_connections 2024-02-19T22:10:47.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:47.494+0000 7f6d9bf94700 1 --2- 172.21.15.88:0/3697536149 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6d940f5410 0x7f6d940f5830 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:47.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:47.494+0000 7f6d9bf94700 1 -- 172.21.15.88:0/3697536149 >> 172.21.15.88:0/3697536149 conn(0x7f6d940f0b70 msgr2=0x7f6d940f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:47.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:47.494+0000 7f6d9bf94700 1 -- 172.21.15.88:0/3697536149 shutdown_connections 2024-02-19T22:10:47.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:47.494+0000 7f6d9bf94700 1 -- 172.21.15.88:0/3697536149 wait complete. 2024-02-19T22:10:47.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:47.495+0000 7f6d9bf94700 1 Processor -- start 2024-02-19T22:10:47.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:47.495+0000 7f6d9bf94700 1 -- start start 2024-02-19T22:10:47.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:47.495+0000 7f6d9bf94700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6d940f5410 0x7f6d941056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:47.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:47.495+0000 7f6d9bf94700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d94105bd0 con 0x7f6d940f5410 2024-02-19T22:10:47.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:47.495+0000 7f6d99d30700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6d940f5410 0x7f6d941056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:47.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:47.495+0000 7f6d99d30700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6d940f5410 0x7f6d941056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:53956/0 (socket says 172.21.15.88:53956) 2024-02-19T22:10:47.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:47.495+0000 7f6d99d30700 1 -- 172.21.15.88:0/4027862395 learned_addr learned my addr 172.21.15.88:0/4027862395 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:10:47.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:47.496+0000 7f6d99d30700 1 -- 172.21.15.88:0/4027862395 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d88005040 con 0x7f6d940f5410 2024-02-19T22:10:47.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:47.496+0000 7f6d99d30700 1 --2- 172.21.15.88:0/4027862395 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6d940f5410 0x7f6d941056e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f6d88005010 tx=0x7f6d8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:47.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:47.496+0000 7f6d867fc700 1 -- 172.21.15.88:0/4027862395 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d88009320 con 0x7f6d940f5410 2024-02-19T22:10:47.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:47.496+0000 7f6d9bf94700 1 -- 172.21.15.88:0/4027862395 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d94105dd0 con 0x7f6d940f5410 2024-02-19T22:10:47.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:47.496+0000 7f6d867fc700 1 -- 172.21.15.88:0/4027862395 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6d88013380 con 0x7f6d940f5410 2024-02-19T22:10:47.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:47.496+0000 7f6d867fc700 1 -- 172.21.15.88:0/4027862395 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d88014e50 con 0x7f6d940f5410 2024-02-19T22:10:47.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:47.496+0000 7f6d9bf94700 1 -- 172.21.15.88:0/4027862395 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d94106260 con 0x7f6d940f5410 2024-02-19T22:10:47.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:47.497+0000 7f6d867fc700 1 -- 172.21.15.88:0/4027862395 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6d88014420 con 0x7f6d940f5410 2024-02-19T22:10:47.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:47.497+0000 7f6d9bf94700 1 -- 172.21.15.88:0/4027862395 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d94045ad0 con 0x7f6d940f5410 2024-02-19T22:10:47.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:47.497+0000 7f6d867fc700 1 --2- 172.21.15.88:0/4027862395 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f6d8002ff10 0x7f6d800323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:47.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:47.497+0000 7f6d867fc700 1 -- 172.21.15.88:0/4027862395 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6d88040e60 con 0x7f6d940f5410 2024-02-19T22:10:47.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:47.497+0000 7f6d9952f700 1 --2- 172.21.15.88:0/4027862395 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f6d8002ff10 0x7f6d800323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:47.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:47.497+0000 7f6d9952f700 1 --2- 172.21.15.88:0/4027862395 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f6d8002ff10 0x7f6d800323d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f6d900096a0 tx=0x7f6d90006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:47.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:47.502+0000 7f6d867fc700 1 -- 172.21.15.88:0/4027862395 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6d880095f0 con 0x7f6d940f5410 2024-02-19T22:10:47.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:47.710+0000 7f6d9bf94700 1 -- 172.21.15.88:0/4027862395 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6d941070b0 con 0x7f6d940f5410 2024-02-19T22:10:47.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:47.711+0000 7f6d867fc700 1 -- 172.21.15.88:0/4027862395 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6d8801db70 con 0x7f6d940f5410 2024-02-19T22:10:47.880 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:10:47.881 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:10:47.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:47.713+0000 7f6d9bf94700 1 -- 172.21.15.88:0/4027862395 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f6d8002ff10 msgr2=0x7f6d800323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:47.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:47.713+0000 7f6d9bf94700 1 --2- 172.21.15.88:0/4027862395 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f6d8002ff10 0x7f6d800323d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f6d900096a0 tx=0x7f6d90006b40).stop 2024-02-19T22:10:47.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:47.713+0000 7f6d9bf94700 1 -- 172.21.15.88:0/4027862395 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6d940f5410 msgr2=0x7f6d941056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:47.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:47.713+0000 7f6d9bf94700 1 --2- 172.21.15.88:0/4027862395 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6d940f5410 0x7f6d941056e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f6d88005010 tx=0x7f6d8800b040).stop 2024-02-19T22:10:47.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:47.713+0000 7f6d9bf94700 1 -- 172.21.15.88:0/4027862395 shutdown_connections 2024-02-19T22:10:47.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:47.713+0000 7f6d9bf94700 1 --2- 172.21.15.88:0/4027862395 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f6d8002ff10 0x7f6d800323d0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:47.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:47.713+0000 7f6d9bf94700 1 --2- 172.21.15.88:0/4027862395 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6d940f5410 0x7f6d941056e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:47.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:47.713+0000 7f6d9bf94700 1 -- 172.21.15.88:0/4027862395 >> 172.21.15.88:0/4027862395 conn(0x7f6d940f0b70 msgr2=0x7f6d940f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:47.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:47.713+0000 7f6d9bf94700 1 -- 172.21.15.88:0/4027862395 shutdown_connections 2024-02-19T22:10:47.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:47.713+0000 7f6d9bf94700 1 -- 172.21.15.88:0/4027862395 wait complete. 2024-02-19T22:10:47.881 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:10:48.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:10:48 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:10:47.711269+0000 mon.smithi052 (mon.0) 127 : audit [DBG] from='client.? 172.21.15.88:0/4027862395' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:10:49.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:10:49 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:10:47.620875+0000 mgr.smithi052.ghqrtk (mgr.14162) 33 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:51.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:10:51 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:10:49.621298+0000 mgr.smithi052.ghqrtk (mgr.14162) 34 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:53.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:10:53 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:10:51.621690+0000 mgr.smithi052.ghqrtk (mgr.14162) 35 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:54.121 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:10:54.121 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:10:55.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:10:55 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:10:53.622017+0000 mgr.smithi052.ghqrtk (mgr.14162) 36 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:55.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:55.935+0000 7fdce8444700 1 -- 172.21.15.88:0/3730475089 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdce00f5410 msgr2=0x7fdce00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:55.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:55.935+0000 7fdce8444700 1 --2- 172.21.15.88:0/3730475089 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdce00f5410 0x7fdce00f5830 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fdcd0009fb0 tx=0x7fdcd0009b30).stop 2024-02-19T22:10:55.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:55.935+0000 7fdce8444700 1 -- 172.21.15.88:0/3730475089 shutdown_connections 2024-02-19T22:10:55.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:55.935+0000 7fdce8444700 1 --2- 172.21.15.88:0/3730475089 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdce00f5410 0x7fdce00f5830 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:55.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:55.935+0000 7fdce8444700 1 -- 172.21.15.88:0/3730475089 >> 172.21.15.88:0/3730475089 conn(0x7fdce00f0b70 msgr2=0x7fdce00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:55.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:55.935+0000 7fdce8444700 1 -- 172.21.15.88:0/3730475089 shutdown_connections 2024-02-19T22:10:55.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:55.935+0000 7fdce8444700 1 -- 172.21.15.88:0/3730475089 wait complete. 2024-02-19T22:10:55.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:55.936+0000 7fdce8444700 1 Processor -- start 2024-02-19T22:10:55.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:55.936+0000 7fdce8444700 1 -- start start 2024-02-19T22:10:55.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:55.936+0000 7fdce8444700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdce00f5410 0x7fdce00fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:55.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:55.936+0000 7fdce8444700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdce00fb960 con 0x7fdce00f5410 2024-02-19T22:10:55.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:55.936+0000 7fdce61e0700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdce00f5410 0x7fdce00fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:55.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:55.936+0000 7fdce61e0700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdce00f5410 0x7fdce00fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:52406/0 (socket says 172.21.15.88:52406) 2024-02-19T22:10:55.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:55.936+0000 7fdce61e0700 1 -- 172.21.15.88:0/3215054437 learned_addr learned my addr 172.21.15.88:0/3215054437 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:10:55.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:55.936+0000 7fdce61e0700 1 -- 172.21.15.88:0/3215054437 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd0005040 con 0x7fdce00f5410 2024-02-19T22:10:55.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:55.936+0000 7fdce61e0700 1 --2- 172.21.15.88:0/3215054437 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdce00f5410 0x7fdce00fd4e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fdcd0005010 tx=0x7fdcd000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:55.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:55.937+0000 7fdcd6ffd700 1 -- 172.21.15.88:0/3215054437 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdcd0009320 con 0x7fdce00f5410 2024-02-19T22:10:55.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:55.937+0000 7fdce8444700 1 -- 172.21.15.88:0/3215054437 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdce00fbb00 con 0x7fdce00f5410 2024-02-19T22:10:55.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:55.937+0000 7fdcd6ffd700 1 -- 172.21.15.88:0/3215054437 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdcd0013380 con 0x7fdce00f5410 2024-02-19T22:10:55.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:55.937+0000 7fdcd6ffd700 1 -- 172.21.15.88:0/3215054437 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdcd0014e50 con 0x7fdce00f5410 2024-02-19T22:10:55.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:55.937+0000 7fdce8444700 1 -- 172.21.15.88:0/3215054437 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdce00fbf90 con 0x7fdce00f5410 2024-02-19T22:10:55.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:55.937+0000 7fdcd6ffd700 1 -- 172.21.15.88:0/3215054437 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdcd00263a0 con 0x7fdce00f5410 2024-02-19T22:10:55.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:55.938+0000 7fdcd6ffd700 1 --2- 172.21.15.88:0/3215054437 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdccc02ff10 0x7fdccc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:55.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:55.938+0000 7fdcd6ffd700 1 -- 172.21.15.88:0/3215054437 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdcd0040f70 con 0x7fdce00f5410 2024-02-19T22:10:55.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:55.938+0000 7fdce8444700 1 -- 172.21.15.88:0/3215054437 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdce0059100 con 0x7fdce00f5410 2024-02-19T22:10:55.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:55.938+0000 7fdce59df700 1 --2- 172.21.15.88:0/3215054437 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdccc02ff10 0x7fdccc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:55.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:55.938+0000 7fdce59df700 1 --2- 172.21.15.88:0/3215054437 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdccc02ff10 0x7fdccc0323d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fdcdc0096a0 tx=0x7fdcdc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:55.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:55.943+0000 7fdcd6ffd700 1 -- 172.21.15.88:0/3215054437 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdcd0012920 con 0x7fdce00f5410 2024-02-19T22:10:56.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:56.146+0000 7fdce8444700 1 -- 172.21.15.88:0/3215054437 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdce0045ad0 con 0x7fdce00f5410 2024-02-19T22:10:56.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:56.147+0000 7fdcd6ffd700 1 -- 172.21.15.88:0/3215054437 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdcd002b020 con 0x7fdce00f5410 2024-02-19T22:10:56.148 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:10:56.148 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:10:56.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:56.149+0000 7fdce8444700 1 -- 172.21.15.88:0/3215054437 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdccc02ff10 msgr2=0x7fdccc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:56.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:56.149+0000 7fdce8444700 1 --2- 172.21.15.88:0/3215054437 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdccc02ff10 0x7fdccc0323d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fdcdc0096a0 tx=0x7fdcdc006b40).stop 2024-02-19T22:10:56.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:56.149+0000 7fdce8444700 1 -- 172.21.15.88:0/3215054437 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdce00f5410 msgr2=0x7fdce00fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:56.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:56.149+0000 7fdce8444700 1 --2- 172.21.15.88:0/3215054437 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdce00f5410 0x7fdce00fd4e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fdcd0005010 tx=0x7fdcd000b040).stop 2024-02-19T22:10:56.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:56.149+0000 7fdce8444700 1 -- 172.21.15.88:0/3215054437 shutdown_connections 2024-02-19T22:10:56.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:56.149+0000 7fdce8444700 1 --2- 172.21.15.88:0/3215054437 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdccc02ff10 0x7fdccc0323d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:56.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:56.149+0000 7fdce8444700 1 --2- 172.21.15.88:0/3215054437 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdce00f5410 0x7fdce00fd4e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:56.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:56.149+0000 7fdce8444700 1 -- 172.21.15.88:0/3215054437 >> 172.21.15.88:0/3215054437 conn(0x7fdce00f0b70 msgr2=0x7fdce005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:56.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:56.149+0000 7fdce8444700 1 -- 172.21.15.88:0/3215054437 shutdown_connections 2024-02-19T22:10:56.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:56.149+0000 7fdce8444700 1 -- 172.21.15.88:0/3215054437 wait complete. 2024-02-19T22:10:56.150 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:10:57.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:10:57 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:10:55.622434+0000 mgr.smithi052.ghqrtk (mgr.14162) 37 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:57.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:10:57 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:10:56.147634+0000 mon.smithi052 (mon.0) 128 : audit [DBG] from='client.? 172.21.15.88:0/3215054437' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:10:57.793 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:10:57.794 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:10:59.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:10:59 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:10:57.622866+0000 mgr.smithi052.ghqrtk (mgr.14162) 38 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:59.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:59.873+0000 7f4859bc6700 1 -- 172.21.15.88:0/1505497482 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4854060280 msgr2=0x7f48540f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:59.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:59.873+0000 7f4859bc6700 1 --2- 172.21.15.88:0/1505497482 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4854060280 0x7f48540f90f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f483c009fb0 tx=0x7f483c009b30).stop 2024-02-19T22:10:59.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:59.873+0000 7f4859bc6700 1 -- 172.21.15.88:0/1505497482 shutdown_connections 2024-02-19T22:10:59.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:59.873+0000 7f4859bc6700 1 --2- 172.21.15.88:0/1505497482 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4854060280 0x7f48540f90f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:59.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:59.873+0000 7f4859bc6700 1 -- 172.21.15.88:0/1505497482 >> 172.21.15.88:0/1505497482 conn(0x7f48540f0b70 msgr2=0x7f48540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:59.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:59.873+0000 7f4859bc6700 1 -- 172.21.15.88:0/1505497482 shutdown_connections 2024-02-19T22:10:59.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:59.873+0000 7f4859bc6700 1 -- 172.21.15.88:0/1505497482 wait complete. 2024-02-19T22:10:59.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:59.874+0000 7f4859bc6700 1 Processor -- start 2024-02-19T22:10:59.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:59.874+0000 7f4859bc6700 1 -- start start 2024-02-19T22:10:59.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:59.874+0000 7f4859bc6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4854060280 0x7f4854106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:59.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:59.874+0000 7f4859bc6700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4854106c70 con 0x7f4854060280 2024-02-19T22:10:59.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:59.875+0000 7f48537fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4854060280 0x7f4854106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:59.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:59.875+0000 7f48537fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4854060280 0x7f4854106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:52420/0 (socket says 172.21.15.88:52420) 2024-02-19T22:10:59.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:59.875+0000 7f48537fe700 1 -- 172.21.15.88:0/943197569 learned_addr learned my addr 172.21.15.88:0/943197569 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:10:59.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:59.875+0000 7f48537fe700 1 -- 172.21.15.88:0/943197569 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f483c005040 con 0x7f4854060280 2024-02-19T22:10:59.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:59.875+0000 7f48537fe700 1 --2- 172.21.15.88:0/943197569 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4854060280 0x7f4854106780 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f483c009830 tx=0x7f483c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:59.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:59.875+0000 7f484bfff700 1 -- 172.21.15.88:0/943197569 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f483c009320 con 0x7f4854060280 2024-02-19T22:10:59.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:59.875+0000 7f4859bc6700 1 -- 172.21.15.88:0/943197569 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4854106e70 con 0x7f4854060280 2024-02-19T22:10:59.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:59.875+0000 7f484bfff700 1 -- 172.21.15.88:0/943197569 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f483c013380 con 0x7f4854060280 2024-02-19T22:10:59.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:59.875+0000 7f484bfff700 1 -- 172.21.15.88:0/943197569 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f483c014420 con 0x7f4854060280 2024-02-19T22:10:59.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:59.875+0000 7f4859bc6700 1 -- 172.21.15.88:0/943197569 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4854107300 con 0x7f4854060280 2024-02-19T22:10:59.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:59.876+0000 7f484bfff700 1 -- 172.21.15.88:0/943197569 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f483c014580 con 0x7f4854060280 2024-02-19T22:10:59.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:59.876+0000 7f4859bc6700 1 -- 172.21.15.88:0/943197569 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4854045ad0 con 0x7f4854060280 2024-02-19T22:10:59.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:59.876+0000 7f484bfff700 1 --2- 172.21.15.88:0/943197569 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f484002ff60 0x7f4840032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:59.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:59.876+0000 7f484bfff700 1 -- 172.21.15.88:0/943197569 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f483c041120 con 0x7f4854060280 2024-02-19T22:10:59.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:59.876+0000 7f4852ffd700 1 --2- 172.21.15.88:0/943197569 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f484002ff60 0x7f4840032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:59.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:59.877+0000 7f4852ffd700 1 --2- 172.21.15.88:0/943197569 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f484002ff60 0x7f4840032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f48440096a0 tx=0x7f4844006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:59.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:10:59.881+0000 7f484bfff700 1 -- 172.21.15.88:0/943197569 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f483c01c320 con 0x7f4854060280 2024-02-19T22:11:00.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:00.092+0000 7f4859bc6700 1 -- 172.21.15.88:0/943197569 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f48540fc090 con 0x7f4854060280 2024-02-19T22:11:00.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:00.092+0000 7f484bfff700 1 -- 172.21.15.88:0/943197569 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f483c004490 con 0x7f4854060280 2024-02-19T22:11:00.094 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:11:00.094 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:11:00.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:00.094+0000 7f4859bc6700 1 -- 172.21.15.88:0/943197569 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f484002ff60 msgr2=0x7f4840032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:00.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:00.094+0000 7f4859bc6700 1 --2- 172.21.15.88:0/943197569 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f484002ff60 0x7f4840032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f48440096a0 tx=0x7f4844006b40).stop 2024-02-19T22:11:00.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:00.094+0000 7f4859bc6700 1 -- 172.21.15.88:0/943197569 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4854060280 msgr2=0x7f4854106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:00.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:00.094+0000 7f4859bc6700 1 --2- 172.21.15.88:0/943197569 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4854060280 0x7f4854106780 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f483c009830 tx=0x7f483c00b040).stop 2024-02-19T22:11:00.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:00.094+0000 7f4859bc6700 1 -- 172.21.15.88:0/943197569 shutdown_connections 2024-02-19T22:11:00.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:00.094+0000 7f4859bc6700 1 --2- 172.21.15.88:0/943197569 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f484002ff60 0x7f4840032420 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:00.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:00.094+0000 7f4859bc6700 1 --2- 172.21.15.88:0/943197569 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4854060280 0x7f4854106780 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:00.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:00.094+0000 7f4859bc6700 1 -- 172.21.15.88:0/943197569 >> 172.21.15.88:0/943197569 conn(0x7f48540f0b70 msgr2=0x7f48540f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:00.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:00.094+0000 7f4859bc6700 1 -- 172.21.15.88:0/943197569 shutdown_connections 2024-02-19T22:11:00.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:00.094+0000 7f4859bc6700 1 -- 172.21.15.88:0/943197569 wait complete. 2024-02-19T22:11:00.097 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:11:00.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:00 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:11:00.093882+0000 mon.smithi052 (mon.0) 129 : audit [DBG] from='client.? 172.21.15.88:0/943197569' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:11:01.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:01 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:10:59.623310+0000 mgr.smithi052.ghqrtk (mgr.14162) 39 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:01.659 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:11:01.660 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:11:03.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:03.540+0000 7f42661a9700 1 -- 172.21.15.88:0/3608957098 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f42600f51a0 msgr2=0x7f42600f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:03.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:03.540+0000 7f42661a9700 1 --2- 172.21.15.88:0/3608957098 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f42600f51a0 0x7f42600f55c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f425c009fb0 tx=0x7f425c009b30).stop 2024-02-19T22:11:03.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:03.541+0000 7f42661a9700 1 -- 172.21.15.88:0/3608957098 shutdown_connections 2024-02-19T22:11:03.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:03.541+0000 7f42661a9700 1 --2- 172.21.15.88:0/3608957098 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f42600f51a0 0x7f42600f55c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:03.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:03.541+0000 7f42661a9700 1 -- 172.21.15.88:0/3608957098 >> 172.21.15.88:0/3608957098 conn(0x7f42600f09a0 msgr2=0x7f42600f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:03.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:03.541+0000 7f42661a9700 1 -- 172.21.15.88:0/3608957098 shutdown_connections 2024-02-19T22:11:03.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:03.541+0000 7f42661a9700 1 -- 172.21.15.88:0/3608957098 wait complete. 2024-02-19T22:11:03.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:03.541+0000 7f42661a9700 1 Processor -- start 2024-02-19T22:11:03.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:03.541+0000 7f42661a9700 1 -- start start 2024-02-19T22:11:03.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:03.541+0000 7f42661a9700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f42600f51a0 0x7f4260109290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:03.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:03.541+0000 7f42661a9700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4260060280 con 0x7f42600f51a0 2024-02-19T22:11:03.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:03.542+0000 7f42651a7700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f42600f51a0 0x7f4260109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:03.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:03.542+0000 7f42651a7700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f42600f51a0 0x7f4260109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:38430/0 (socket says 172.21.15.88:38430) 2024-02-19T22:11:03.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:03.542+0000 7f42651a7700 1 -- 172.21.15.88:0/3292317606 learned_addr learned my addr 172.21.15.88:0/3292317606 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:11:03.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:03.542+0000 7f42651a7700 1 -- 172.21.15.88:0/3292317606 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f425c005040 con 0x7f42600f51a0 2024-02-19T22:11:03.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:03.542+0000 7f42651a7700 1 --2- 172.21.15.88:0/3292317606 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f42600f51a0 0x7f4260109290 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f425c0089f0 tx=0x7f425c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:03.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:03.542+0000 7f4255ffb700 1 -- 172.21.15.88:0/3292317606 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f425c009320 con 0x7f42600f51a0 2024-02-19T22:11:03.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:03.542+0000 7f42661a9700 1 -- 172.21.15.88:0/3292317606 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4260060480 con 0x7f42600f51a0 2024-02-19T22:11:03.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:03.542+0000 7f4255ffb700 1 -- 172.21.15.88:0/3292317606 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f425c013380 con 0x7f42600f51a0 2024-02-19T22:11:03.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:03.542+0000 7f4255ffb700 1 -- 172.21.15.88:0/3292317606 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f425c014e20 con 0x7f42600f51a0 2024-02-19T22:11:03.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:03.542+0000 7f42661a9700 1 -- 172.21.15.88:0/3292317606 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42601099c0 con 0x7f42600f51a0 2024-02-19T22:11:03.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:03.543+0000 7f4255ffb700 1 -- 172.21.15.88:0/3292317606 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f425c0095f0 con 0x7f42600f51a0 2024-02-19T22:11:03.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:03.543+0000 7f42661a9700 1 -- 172.21.15.88:0/3292317606 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4260045ad0 con 0x7f42600f51a0 2024-02-19T22:11:03.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:03.543+0000 7f4255ffb700 1 --2- 172.21.15.88:0/3292317606 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f424c02ff60 0x7f424c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:03.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:03.543+0000 7f4255ffb700 1 -- 172.21.15.88:0/3292317606 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f425c040f20 con 0x7f42600f51a0 2024-02-19T22:11:03.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:03.544+0000 7f42649a6700 1 --2- 172.21.15.88:0/3292317606 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f424c02ff60 0x7f424c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:03.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:03.544+0000 7f42649a6700 1 --2- 172.21.15.88:0/3292317606 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f424c02ff60 0x7f424c032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f42500096a0 tx=0x7f4250006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:03.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:03.548+0000 7f4255ffb700 1 -- 172.21.15.88:0/3292317606 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f425c014420 con 0x7f42600f51a0 2024-02-19T22:11:03.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:03 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:11:01.623657+0000 mgr.smithi052.ghqrtk (mgr.14162) 40 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:03.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:03.752+0000 7f42661a9700 1 -- 172.21.15.88:0/3292317606 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f42600fad30 con 0x7f42600f51a0 2024-02-19T22:11:03.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:03.753+0000 7f4255ffb700 1 -- 172.21.15.88:0/3292317606 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f425c010230 con 0x7f42600f51a0 2024-02-19T22:11:03.754 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:11:03.754 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:11:03.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:03.754+0000 7f42661a9700 1 -- 172.21.15.88:0/3292317606 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f424c02ff60 msgr2=0x7f424c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:03.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:03.754+0000 7f42661a9700 1 --2- 172.21.15.88:0/3292317606 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f424c02ff60 0x7f424c032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f42500096a0 tx=0x7f4250006b40).stop 2024-02-19T22:11:03.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:03.754+0000 7f42661a9700 1 -- 172.21.15.88:0/3292317606 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f42600f51a0 msgr2=0x7f4260109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:03.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:03.754+0000 7f42661a9700 1 --2- 172.21.15.88:0/3292317606 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f42600f51a0 0x7f4260109290 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f425c0089f0 tx=0x7f425c00b040).stop 2024-02-19T22:11:03.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:03.754+0000 7f42661a9700 1 -- 172.21.15.88:0/3292317606 shutdown_connections 2024-02-19T22:11:03.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:03.754+0000 7f42661a9700 1 --2- 172.21.15.88:0/3292317606 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f424c02ff60 0x7f424c032420 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:03.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:03.754+0000 7f42661a9700 1 --2- 172.21.15.88:0/3292317606 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f42600f51a0 0x7f4260109290 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:03.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:03.754+0000 7f42661a9700 1 -- 172.21.15.88:0/3292317606 >> 172.21.15.88:0/3292317606 conn(0x7f42600f09a0 msgr2=0x7f42600f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:03.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:03.754+0000 7f42661a9700 1 -- 172.21.15.88:0/3292317606 shutdown_connections 2024-02-19T22:11:03.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:03.754+0000 7f42661a9700 1 -- 172.21.15.88:0/3292317606 wait complete. 2024-02-19T22:11:03.757 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:11:04.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:04 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:11:03.754013+0000 mon.smithi052 (mon.0) 130 : audit [DBG] from='client.? 172.21.15.88:0/3292317606' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:11:05.496 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:11:05.496 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:11:05.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:05 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:11:03.624048+0000 mgr.smithi052.ghqrtk (mgr.14162) 41 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:07.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:07.364+0000 7f1fc3fff700 1 -- 172.21.15.88:0/2993613040 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1fc40f51a0 msgr2=0x7f1fc40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:07.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:07.364+0000 7f1fc3fff700 1 --2- 172.21.15.88:0/2993613040 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1fc40f51a0 0x7f1fc40f55c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f1fb4009fb0 tx=0x7f1fb4009b30).stop 2024-02-19T22:11:07.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:07.364+0000 7f1fc3fff700 1 -- 172.21.15.88:0/2993613040 shutdown_connections 2024-02-19T22:11:07.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:07.364+0000 7f1fc3fff700 1 --2- 172.21.15.88:0/2993613040 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1fc40f51a0 0x7f1fc40f55c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:07.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:07.364+0000 7f1fc3fff700 1 -- 172.21.15.88:0/2993613040 >> 172.21.15.88:0/2993613040 conn(0x7f1fc40f09a0 msgr2=0x7f1fc40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:07.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:07.364+0000 7f1fc3fff700 1 -- 172.21.15.88:0/2993613040 shutdown_connections 2024-02-19T22:11:07.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:07.364+0000 7f1fc3fff700 1 -- 172.21.15.88:0/2993613040 wait complete. 2024-02-19T22:11:07.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:07.365+0000 7f1fc3fff700 1 Processor -- start 2024-02-19T22:11:07.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:07.365+0000 7f1fc3fff700 1 -- start start 2024-02-19T22:11:07.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:07.365+0000 7f1fc3fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1fc40f51a0 0x7f1fc41065e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:07.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:07.365+0000 7f1fc3fff700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc4106ad0 con 0x7f1fc40f51a0 2024-02-19T22:11:07.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:07.365+0000 7f1fc2ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1fc40f51a0 0x7f1fc41065e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:07.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:07.365+0000 7f1fc2ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1fc40f51a0 0x7f1fc41065e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:38448/0 (socket says 172.21.15.88:38448) 2024-02-19T22:11:07.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:07.365+0000 7f1fc2ffd700 1 -- 172.21.15.88:0/3278708754 learned_addr learned my addr 172.21.15.88:0/3278708754 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:11:07.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:07.366+0000 7f1fc2ffd700 1 -- 172.21.15.88:0/3278708754 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb4005040 con 0x7f1fc40f51a0 2024-02-19T22:11:07.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:07.366+0000 7f1fc2ffd700 1 --2- 172.21.15.88:0/3278708754 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1fc40f51a0 0x7f1fc41065e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f1fb40089f0 tx=0x7f1fb400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:07.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:07.366+0000 7f1fab7fe700 1 -- 172.21.15.88:0/3278708754 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fb4009320 con 0x7f1fc40f51a0 2024-02-19T22:11:07.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:07.366+0000 7f1fc3fff700 1 -- 172.21.15.88:0/3278708754 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fc4106cd0 con 0x7f1fc40f51a0 2024-02-19T22:11:07.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:07.366+0000 7f1fab7fe700 1 -- 172.21.15.88:0/3278708754 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1fb4013380 con 0x7f1fc40f51a0 2024-02-19T22:11:07.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:07.366+0000 7f1fab7fe700 1 -- 172.21.15.88:0/3278708754 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fb4014e20 con 0x7f1fc40f51a0 2024-02-19T22:11:07.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:07.366+0000 7f1fc3fff700 1 -- 172.21.15.88:0/3278708754 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fc4107160 con 0x7f1fc40f51a0 2024-02-19T22:11:07.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:07.367+0000 7f1fab7fe700 1 -- 172.21.15.88:0/3278708754 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1fb4014420 con 0x7f1fc40f51a0 2024-02-19T22:11:07.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:07.367+0000 7f1fc3fff700 1 -- 172.21.15.88:0/3278708754 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fc4045ad0 con 0x7f1fc40f51a0 2024-02-19T22:11:07.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:07.367+0000 7f1fab7fe700 1 --2- 172.21.15.88:0/3278708754 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f1fac02ff10 0x7f1fac0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:07.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:07.367+0000 7f1fab7fe700 1 -- 172.21.15.88:0/3278708754 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1fb4040ed0 con 0x7f1fc40f51a0 2024-02-19T22:11:07.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:07.367+0000 7f1fc27fc700 1 --2- 172.21.15.88:0/3278708754 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f1fac02ff10 0x7f1fac0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:07.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:07.368+0000 7f1fc27fc700 1 --2- 172.21.15.88:0/3278708754 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f1fac02ff10 0x7f1fac0323d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f1fb80096a0 tx=0x7f1fb8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:07.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:07.372+0000 7f1fab7fe700 1 -- 172.21.15.88:0/3278708754 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1fb4014790 con 0x7f1fc40f51a0 2024-02-19T22:11:07.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:07.578+0000 7f1fc3fff700 1 -- 172.21.15.88:0/3278708754 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1fc4107fb0 con 0x7f1fc40f51a0 2024-02-19T22:11:07.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:07.579+0000 7f1fab7fe700 1 -- 172.21.15.88:0/3278708754 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1fb401d7f0 con 0x7f1fc40f51a0 2024-02-19T22:11:07.581 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:11:07.581 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:11:07.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:07.581+0000 7f1fc3fff700 1 -- 172.21.15.88:0/3278708754 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f1fac02ff10 msgr2=0x7f1fac0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:07.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:07.581+0000 7f1fc3fff700 1 --2- 172.21.15.88:0/3278708754 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f1fac02ff10 0x7f1fac0323d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f1fb80096a0 tx=0x7f1fb8006b40).stop 2024-02-19T22:11:07.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:07.581+0000 7f1fc3fff700 1 -- 172.21.15.88:0/3278708754 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1fc40f51a0 msgr2=0x7f1fc41065e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:07.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:07.581+0000 7f1fc3fff700 1 --2- 172.21.15.88:0/3278708754 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1fc40f51a0 0x7f1fc41065e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f1fb40089f0 tx=0x7f1fb400b040).stop 2024-02-19T22:11:07.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:07.581+0000 7f1fc3fff700 1 -- 172.21.15.88:0/3278708754 shutdown_connections 2024-02-19T22:11:07.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:07.581+0000 7f1fc3fff700 1 --2- 172.21.15.88:0/3278708754 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f1fac02ff10 0x7f1fac0323d0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:07.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:07.581+0000 7f1fc3fff700 1 --2- 172.21.15.88:0/3278708754 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1fc40f51a0 0x7f1fc41065e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:07.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:07.581+0000 7f1fc3fff700 1 -- 172.21.15.88:0/3278708754 >> 172.21.15.88:0/3278708754 conn(0x7f1fc40f09a0 msgr2=0x7f1fc40f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:07.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:07.581+0000 7f1fc3fff700 1 -- 172.21.15.88:0/3278708754 shutdown_connections 2024-02-19T22:11:07.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:07.581+0000 7f1fc3fff700 1 -- 172.21.15.88:0/3278708754 wait complete. 2024-02-19T22:11:07.583 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:11:07.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:07 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:11:05.624476+0000 mgr.smithi052.ghqrtk (mgr.14162) 42 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:08.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:08 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:11:07.580554+0000 mon.smithi052 (mon.0) 131 : audit [DBG] from='client.? 172.21.15.88:0/3278708754' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:11:09.152 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:11:09.152 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:11:09.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:09 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:11:07.624809+0000 mgr.smithi052.ghqrtk (mgr.14162) 43 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:10.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:10.912+0000 7f220d2cc700 1 -- 172.21.15.88:0/816832447 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f22080f6510 msgr2=0x7f22080f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:10.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:10.912+0000 7f220d2cc700 1 --2- 172.21.15.88:0/816832447 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f22080f6510 0x7f22080f6930 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f21fc009fb0 tx=0x7f21fc009b30).stop 2024-02-19T22:11:10.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:10.912+0000 7f220d2cc700 1 -- 172.21.15.88:0/816832447 shutdown_connections 2024-02-19T22:11:10.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:10.912+0000 7f220d2cc700 1 --2- 172.21.15.88:0/816832447 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f22080f6510 0x7f22080f6930 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:10.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:10.912+0000 7f220d2cc700 1 -- 172.21.15.88:0/816832447 >> 172.21.15.88:0/816832447 conn(0x7f22080f1c70 msgr2=0x7f22080f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:10.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:10.912+0000 7f220d2cc700 1 -- 172.21.15.88:0/816832447 shutdown_connections 2024-02-19T22:11:10.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:10.912+0000 7f220d2cc700 1 -- 172.21.15.88:0/816832447 wait complete. 2024-02-19T22:11:10.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:10.913+0000 7f220d2cc700 1 Processor -- start 2024-02-19T22:11:10.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:10.913+0000 7f220d2cc700 1 -- start start 2024-02-19T22:11:10.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:10.913+0000 7f220d2cc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f22080f6510 0x7f22081067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:10.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:10.913+0000 7f220d2cc700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2208106cd0 con 0x7f22080f6510 2024-02-19T22:11:10.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:10.913+0000 7f2206ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f22080f6510 0x7f22081067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:10.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:10.913+0000 7f2206ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f22080f6510 0x7f22081067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:38472/0 (socket says 172.21.15.88:38472) 2024-02-19T22:11:10.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:10.913+0000 7f2206ffd700 1 -- 172.21.15.88:0/3830212670 learned_addr learned my addr 172.21.15.88:0/3830212670 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:11:10.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:10.913+0000 7f2206ffd700 1 -- 172.21.15.88:0/3830212670 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21fc005040 con 0x7f22080f6510 2024-02-19T22:11:10.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:10.914+0000 7f2206ffd700 1 --2- 172.21.15.88:0/3830212670 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f22080f6510 0x7f22081067e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f21fc005010 tx=0x7f21fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:10.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:10.914+0000 7f21ef7fe700 1 -- 172.21.15.88:0/3830212670 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21fc009320 con 0x7f22080f6510 2024-02-19T22:11:10.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:10.914+0000 7f220d2cc700 1 -- 172.21.15.88:0/3830212670 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2208106ed0 con 0x7f22080f6510 2024-02-19T22:11:10.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:10.914+0000 7f21ef7fe700 1 -- 172.21.15.88:0/3830212670 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f21fc013380 con 0x7f22080f6510 2024-02-19T22:11:10.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:10.914+0000 7f21ef7fe700 1 -- 172.21.15.88:0/3830212670 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21fc014e50 con 0x7f22080f6510 2024-02-19T22:11:10.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:10.914+0000 7f220d2cc700 1 -- 172.21.15.88:0/3830212670 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2208107360 con 0x7f22080f6510 2024-02-19T22:11:10.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:10.915+0000 7f21ef7fe700 1 -- 172.21.15.88:0/3830212670 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f21fc0095f0 con 0x7f22080f6510 2024-02-19T22:11:10.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:10.915+0000 7f220d2cc700 1 -- 172.21.15.88:0/3830212670 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2208045ad0 con 0x7f22080f6510 2024-02-19T22:11:10.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:10.915+0000 7f21ef7fe700 1 --2- 172.21.15.88:0/3830212670 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f21f002ff10 0x7f21f00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:10.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:10.915+0000 7f21ef7fe700 1 -- 172.21.15.88:0/3830212670 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f21fc040fa0 con 0x7f22080f6510 2024-02-19T22:11:10.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:10.915+0000 7f22067fc700 1 --2- 172.21.15.88:0/3830212670 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f21f002ff10 0x7f21f00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:10.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:10.915+0000 7f22067fc700 1 --2- 172.21.15.88:0/3830212670 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f21f002ff10 0x7f21f00323d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f21f80096a0 tx=0x7f21f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:10.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:10.920+0000 7f21ef7fe700 1 -- 172.21.15.88:0/3830212670 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f21fc014420 con 0x7f22080f6510 2024-02-19T22:11:11.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:11.125+0000 7f220d2cc700 1 -- 172.21.15.88:0/3830212670 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2208108210 con 0x7f22080f6510 2024-02-19T22:11:11.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:11.126+0000 7f21ef7fe700 1 -- 172.21.15.88:0/3830212670 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f21fc010230 con 0x7f22080f6510 2024-02-19T22:11:11.127 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:11:11.127 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:11:11.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:11.127+0000 7f220d2cc700 1 -- 172.21.15.88:0/3830212670 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f21f002ff10 msgr2=0x7f21f00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:11.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:11.127+0000 7f220d2cc700 1 --2- 172.21.15.88:0/3830212670 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f21f002ff10 0x7f21f00323d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f21f80096a0 tx=0x7f21f8006b40).stop 2024-02-19T22:11:11.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:11.127+0000 7f220d2cc700 1 -- 172.21.15.88:0/3830212670 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f22080f6510 msgr2=0x7f22081067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:11.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:11.127+0000 7f220d2cc700 1 --2- 172.21.15.88:0/3830212670 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f22080f6510 0x7f22081067e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f21fc005010 tx=0x7f21fc00b040).stop 2024-02-19T22:11:11.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:11.127+0000 7f220d2cc700 1 -- 172.21.15.88:0/3830212670 shutdown_connections 2024-02-19T22:11:11.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:11.127+0000 7f220d2cc700 1 --2- 172.21.15.88:0/3830212670 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f21f002ff10 0x7f21f00323d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:11.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:11.127+0000 7f220d2cc700 1 --2- 172.21.15.88:0/3830212670 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f22080f6510 0x7f22081067e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:11.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:11.127+0000 7f220d2cc700 1 -- 172.21.15.88:0/3830212670 >> 172.21.15.88:0/3830212670 conn(0x7f22080f1c70 msgr2=0x7f22080f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:11.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:11.127+0000 7f220d2cc700 1 -- 172.21.15.88:0/3830212670 shutdown_connections 2024-02-19T22:11:11.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:11.127+0000 7f220d2cc700 1 -- 172.21.15.88:0/3830212670 wait complete. 2024-02-19T22:11:11.130 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:11:11.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:11 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:11:09.625216+0000 mgr.smithi052.ghqrtk (mgr.14162) 44 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:12.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:12 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:11:11.127207+0000 mon.smithi052 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.88:0/3830212670' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:11:12.747 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:11:12.747 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:11:13.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:13 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:11:11.625643+0000 mgr.smithi052.ghqrtk (mgr.14162) 45 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:14.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:14.536+0000 7f7d0692a700 1 -- 172.21.15.88:0/1466920079 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d00060280 msgr2=0x7f7d000f9010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:14.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:14.536+0000 7f7d0692a700 1 --2- 172.21.15.88:0/1466920079 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d00060280 0x7f7d000f9010 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f7ce8009fb0 tx=0x7f7ce8009b30).stop 2024-02-19T22:11:14.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:14.536+0000 7f7d0692a700 1 -- 172.21.15.88:0/1466920079 shutdown_connections 2024-02-19T22:11:14.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:14.536+0000 7f7d0692a700 1 --2- 172.21.15.88:0/1466920079 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d00060280 0x7f7d000f9010 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:14.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:14.536+0000 7f7d0692a700 1 -- 172.21.15.88:0/1466920079 >> 172.21.15.88:0/1466920079 conn(0x7f7d000f0a90 msgr2=0x7f7d000f2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:14.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:14.536+0000 7f7d0692a700 1 -- 172.21.15.88:0/1466920079 shutdown_connections 2024-02-19T22:11:14.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:14.536+0000 7f7d0692a700 1 -- 172.21.15.88:0/1466920079 wait complete. 2024-02-19T22:11:14.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:14.537+0000 7f7d0692a700 1 Processor -- start 2024-02-19T22:11:14.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:14.537+0000 7f7d0692a700 1 -- start start 2024-02-19T22:11:14.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:14.537+0000 7f7d0692a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d00060280 0x7f7d000f6a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:14.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:14.537+0000 7f7d0692a700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d000f4f10 con 0x7f7d00060280 2024-02-19T22:11:14.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:14.537+0000 7f7cfffff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d00060280 0x7f7d000f6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:14.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:14.537+0000 7f7cfffff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d00060280 0x7f7d000f6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:45048/0 (socket says 172.21.15.88:45048) 2024-02-19T22:11:14.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:14.537+0000 7f7cfffff700 1 -- 172.21.15.88:0/1418074998 learned_addr learned my addr 172.21.15.88:0/1418074998 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:11:14.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:14.538+0000 7f7cfffff700 1 -- 172.21.15.88:0/1418074998 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ce8005040 con 0x7f7d00060280 2024-02-19T22:11:14.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:14.538+0000 7f7cfffff700 1 --2- 172.21.15.88:0/1418074998 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d00060280 0x7f7d000f6a90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f7ce80089f0 tx=0x7f7ce800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:14.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:14.538+0000 7f7cfcff9700 1 -- 172.21.15.88:0/1418074998 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ce8009320 con 0x7f7d00060280 2024-02-19T22:11:14.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:14.538+0000 7f7d0692a700 1 -- 172.21.15.88:0/1418074998 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d000f50b0 con 0x7f7d00060280 2024-02-19T22:11:14.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:14.538+0000 7f7cfcff9700 1 -- 172.21.15.88:0/1418074998 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7ce8013380 con 0x7f7d00060280 2024-02-19T22:11:14.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:14.538+0000 7f7cfcff9700 1 -- 172.21.15.88:0/1418074998 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ce8014e20 con 0x7f7d00060280 2024-02-19T22:11:14.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:14.538+0000 7f7d0692a700 1 -- 172.21.15.88:0/1418074998 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d000f5540 con 0x7f7d00060280 2024-02-19T22:11:14.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:14.539+0000 7f7cfcff9700 1 -- 172.21.15.88:0/1418074998 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7ce80263a0 con 0x7f7d00060280 2024-02-19T22:11:14.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:14.539+0000 7f7cfcff9700 1 --2- 172.21.15.88:0/1418074998 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f7cec02ff10 0x7f7cec0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:14.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:14.539+0000 7f7cfcff9700 1 -- 172.21.15.88:0/1418074998 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7ce8040ed0 con 0x7f7d00060280 2024-02-19T22:11:14.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:14.539+0000 7f7d0692a700 1 -- 172.21.15.88:0/1418074998 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d00045ad0 con 0x7f7d00060280 2024-02-19T22:11:14.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:14.539+0000 7f7cff7fe700 1 --2- 172.21.15.88:0/1418074998 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f7cec02ff10 0x7f7cec0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:14.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:14.539+0000 7f7cff7fe700 1 --2- 172.21.15.88:0/1418074998 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f7cec02ff10 0x7f7cec0323d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f7cf00096a0 tx=0x7f7cf0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:14.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:14.544+0000 7f7cfcff9700 1 -- 172.21.15.88:0/1418074998 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7ce8012920 con 0x7f7d00060280 2024-02-19T22:11:14.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:14.748+0000 7f7d0692a700 1 -- 172.21.15.88:0/1418074998 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7d000f5c60 con 0x7f7d00060280 2024-02-19T22:11:14.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:14.748+0000 7f7cfcff9700 1 -- 172.21.15.88:0/1418074998 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7ce80126f0 con 0x7f7d00060280 2024-02-19T22:11:14.749 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:11:14.750 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:11:14.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:14.750+0000 7f7d0692a700 1 -- 172.21.15.88:0/1418074998 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f7cec02ff10 msgr2=0x7f7cec0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:14.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:14.750+0000 7f7d0692a700 1 --2- 172.21.15.88:0/1418074998 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f7cec02ff10 0x7f7cec0323d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f7cf00096a0 tx=0x7f7cf0006b40).stop 2024-02-19T22:11:14.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:14.750+0000 7f7d0692a700 1 -- 172.21.15.88:0/1418074998 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d00060280 msgr2=0x7f7d000f6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:14.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:14.750+0000 7f7d0692a700 1 --2- 172.21.15.88:0/1418074998 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d00060280 0x7f7d000f6a90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f7ce80089f0 tx=0x7f7ce800b040).stop 2024-02-19T22:11:14.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:14.750+0000 7f7d0692a700 1 -- 172.21.15.88:0/1418074998 shutdown_connections 2024-02-19T22:11:14.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:14.750+0000 7f7d0692a700 1 --2- 172.21.15.88:0/1418074998 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f7cec02ff10 0x7f7cec0323d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:14.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:14.750+0000 7f7d0692a700 1 --2- 172.21.15.88:0/1418074998 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d00060280 0x7f7d000f6a90 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:14.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:14.750+0000 7f7d0692a700 1 -- 172.21.15.88:0/1418074998 >> 172.21.15.88:0/1418074998 conn(0x7f7d000f0a90 msgr2=0x7f7d000f14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:14.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:14.750+0000 7f7d0692a700 1 -- 172.21.15.88:0/1418074998 shutdown_connections 2024-02-19T22:11:14.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:14.750+0000 7f7d0692a700 1 -- 172.21.15.88:0/1418074998 wait complete. 2024-02-19T22:11:14.752 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:11:15.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:15 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:11:13.625972+0000 mgr.smithi052.ghqrtk (mgr.14162) 46 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:15.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:15 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:11:14.749560+0000 mon.smithi052 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.88:0/1418074998' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:11:16.409 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:11:16.409 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:11:17.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:17 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:11:15.626391+0000 mgr.smithi052.ghqrtk (mgr.14162) 47 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:18.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:18.173+0000 7f97fc4cd700 1 -- 172.21.15.88:0/3901561901 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f97f40f6510 msgr2=0x7f97f40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:18.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:18.173+0000 7f97fc4cd700 1 --2- 172.21.15.88:0/3901561901 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f97f40f6510 0x7f97f40f6930 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f97e4009fb0 tx=0x7f97e4009b30).stop 2024-02-19T22:11:18.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:18.173+0000 7f97fc4cd700 1 -- 172.21.15.88:0/3901561901 shutdown_connections 2024-02-19T22:11:18.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:18.173+0000 7f97fc4cd700 1 --2- 172.21.15.88:0/3901561901 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f97f40f6510 0x7f97f40f6930 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:18.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:18.173+0000 7f97fc4cd700 1 -- 172.21.15.88:0/3901561901 >> 172.21.15.88:0/3901561901 conn(0x7f97f40f1c70 msgr2=0x7f97f40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:18.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:18.173+0000 7f97fc4cd700 1 -- 172.21.15.88:0/3901561901 shutdown_connections 2024-02-19T22:11:18.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:18.173+0000 7f97fc4cd700 1 -- 172.21.15.88:0/3901561901 wait complete. 2024-02-19T22:11:18.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:18.173+0000 7f97fc4cd700 1 Processor -- start 2024-02-19T22:11:18.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:18.173+0000 7f97fc4cd700 1 -- start start 2024-02-19T22:11:18.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:18.174+0000 7f97fc4cd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f97f40f6510 0x7f97f41067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:18.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:18.174+0000 7f97fc4cd700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f4106cd0 con 0x7f97f40f6510 2024-02-19T22:11:18.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:18.174+0000 7f97fa269700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f97f40f6510 0x7f97f41067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:18.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:18.174+0000 7f97fa269700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f97f40f6510 0x7f97f41067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:45070/0 (socket says 172.21.15.88:45070) 2024-02-19T22:11:18.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:18.174+0000 7f97fa269700 1 -- 172.21.15.88:0/2007664210 learned_addr learned my addr 172.21.15.88:0/2007664210 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:11:18.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:18.174+0000 7f97fa269700 1 -- 172.21.15.88:0/2007664210 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e4005040 con 0x7f97f40f6510 2024-02-19T22:11:18.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:18.174+0000 7f97fa269700 1 --2- 172.21.15.88:0/2007664210 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f97f40f6510 0x7f97f41067e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f97e4005010 tx=0x7f97e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:18.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:18.174+0000 7f97eaffd700 1 -- 172.21.15.88:0/2007664210 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f97e4009320 con 0x7f97f40f6510 2024-02-19T22:11:18.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:18.174+0000 7f97eaffd700 1 -- 172.21.15.88:0/2007664210 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f97e4013380 con 0x7f97f40f6510 2024-02-19T22:11:18.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:18.174+0000 7f97fc4cd700 1 -- 172.21.15.88:0/2007664210 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97f4106ed0 con 0x7f97f40f6510 2024-02-19T22:11:18.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:18.174+0000 7f97eaffd700 1 -- 172.21.15.88:0/2007664210 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f97e4014e50 con 0x7f97f40f6510 2024-02-19T22:11:18.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:18.174+0000 7f97fc4cd700 1 -- 172.21.15.88:0/2007664210 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97f4107360 con 0x7f97f40f6510 2024-02-19T22:11:18.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:18.175+0000 7f97eaffd700 1 -- 172.21.15.88:0/2007664210 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f97e40095f0 con 0x7f97f40f6510 2024-02-19T22:11:18.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:18.175+0000 7f97fc4cd700 1 -- 172.21.15.88:0/2007664210 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97f4045ad0 con 0x7f97f40f6510 2024-02-19T22:11:18.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:18.175+0000 7f97eaffd700 1 --2- 172.21.15.88:0/2007664210 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f97e002ff10 0x7f97e00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:18.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:18.175+0000 7f97eaffd700 1 -- 172.21.15.88:0/2007664210 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f97e4040e90 con 0x7f97f40f6510 2024-02-19T22:11:18.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:18.176+0000 7f97f9a68700 1 --2- 172.21.15.88:0/2007664210 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f97e002ff10 0x7f97e00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:18.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:18.176+0000 7f97f9a68700 1 --2- 172.21.15.88:0/2007664210 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f97e002ff10 0x7f97e00323d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f97f00096a0 tx=0x7f97f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:18.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:18.180+0000 7f97eaffd700 1 -- 172.21.15.88:0/2007664210 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f97e4014420 con 0x7f97f40f6510 2024-02-19T22:11:18.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:18.385+0000 7f97fc4cd700 1 -- 172.21.15.88:0/2007664210 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f97f4108210 con 0x7f97f40f6510 2024-02-19T22:11:18.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:18.386+0000 7f97eaffd700 1 -- 172.21.15.88:0/2007664210 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f97e4010230 con 0x7f97f40f6510 2024-02-19T22:11:18.388 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:11:18.388 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:11:18.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:18.387+0000 7f97fc4cd700 1 -- 172.21.15.88:0/2007664210 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f97e002ff10 msgr2=0x7f97e00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:18.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:18.387+0000 7f97fc4cd700 1 --2- 172.21.15.88:0/2007664210 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f97e002ff10 0x7f97e00323d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f97f00096a0 tx=0x7f97f0006b40).stop 2024-02-19T22:11:18.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:18.388+0000 7f97fc4cd700 1 -- 172.21.15.88:0/2007664210 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f97f40f6510 msgr2=0x7f97f41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:18.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:18.388+0000 7f97fc4cd700 1 --2- 172.21.15.88:0/2007664210 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f97f40f6510 0x7f97f41067e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f97e4005010 tx=0x7f97e400b040).stop 2024-02-19T22:11:18.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:18.388+0000 7f97fc4cd700 1 -- 172.21.15.88:0/2007664210 shutdown_connections 2024-02-19T22:11:18.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:18.388+0000 7f97fc4cd700 1 --2- 172.21.15.88:0/2007664210 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f97e002ff10 0x7f97e00323d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:18.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:18.388+0000 7f97fc4cd700 1 --2- 172.21.15.88:0/2007664210 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f97f40f6510 0x7f97f41067e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:18.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:18.388+0000 7f97fc4cd700 1 -- 172.21.15.88:0/2007664210 >> 172.21.15.88:0/2007664210 conn(0x7f97f40f1c70 msgr2=0x7f97f40f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:18.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:18.388+0000 7f97fc4cd700 1 -- 172.21.15.88:0/2007664210 shutdown_connections 2024-02-19T22:11:18.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:18.388+0000 7f97fc4cd700 1 -- 172.21.15.88:0/2007664210 wait complete. 2024-02-19T22:11:18.390 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:11:19.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:19 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:11:17.626702+0000 mgr.smithi052.ghqrtk (mgr.14162) 48 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:19.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:19 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:11:18.387550+0000 mon.smithi052 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.88:0/2007664210' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:11:20.013 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:11:20.013 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:11:21.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:21 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:11:19.627120+0000 mgr.smithi052.ghqrtk (mgr.14162) 49 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:21.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:21.784+0000 7f822aeed700 1 -- 172.21.15.88:0/1940854297 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82240f5410 msgr2=0x7f82240f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:21.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:21.784+0000 7f822aeed700 1 --2- 172.21.15.88:0/1940854297 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82240f5410 0x7f82240f5830 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f821c009960 tx=0x7f821c004d10).stop 2024-02-19T22:11:21.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:21.784+0000 7f822aeed700 1 -- 172.21.15.88:0/1940854297 shutdown_connections 2024-02-19T22:11:21.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:21.784+0000 7f822aeed700 1 --2- 172.21.15.88:0/1940854297 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82240f5410 0x7f82240f5830 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:21.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:21.784+0000 7f822aeed700 1 -- 172.21.15.88:0/1940854297 >> 172.21.15.88:0/1940854297 conn(0x7f82240f0b70 msgr2=0x7f82240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:21.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:21.784+0000 7f822aeed700 1 -- 172.21.15.88:0/1940854297 shutdown_connections 2024-02-19T22:11:21.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:21.784+0000 7f822aeed700 1 -- 172.21.15.88:0/1940854297 wait complete. 2024-02-19T22:11:21.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:21.785+0000 7f822aeed700 1 Processor -- start 2024-02-19T22:11:21.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:21.785+0000 7f822aeed700 1 -- start start 2024-02-19T22:11:21.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:21.785+0000 7f822aeed700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82240f5410 0x7f82241067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:21.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:21.785+0000 7f822aeed700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8224106cd0 con 0x7f82240f5410 2024-02-19T22:11:21.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:21.785+0000 7f8228c89700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82240f5410 0x7f82241067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:21.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:21.785+0000 7f8228c89700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82240f5410 0x7f82241067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:45088/0 (socket says 172.21.15.88:45088) 2024-02-19T22:11:21.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:21.785+0000 7f8228c89700 1 -- 172.21.15.88:0/4095442592 learned_addr learned my addr 172.21.15.88:0/4095442592 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:11:21.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:21.786+0000 7f8228c89700 1 -- 172.21.15.88:0/4095442592 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f821c009640 con 0x7f82240f5410 2024-02-19T22:11:21.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:21.786+0000 7f8228c89700 1 --2- 172.21.15.88:0/4095442592 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82240f5410 0x7f82241067e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f821c004b70 tx=0x7f821c004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:21.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:21.786+0000 7f82217fa700 1 -- 172.21.15.88:0/4095442592 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f821c018070 con 0x7f82240f5410 2024-02-19T22:11:21.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:21.786+0000 7f822aeed700 1 -- 172.21.15.88:0/4095442592 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8224106ed0 con 0x7f82240f5410 2024-02-19T22:11:21.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:21.786+0000 7f82217fa700 1 -- 172.21.15.88:0/4095442592 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f821c00b970 con 0x7f82240f5410 2024-02-19T22:11:21.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:21.786+0000 7f82217fa700 1 -- 172.21.15.88:0/4095442592 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f821c01c560 con 0x7f82240f5410 2024-02-19T22:11:21.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:21.786+0000 7f822aeed700 1 -- 172.21.15.88:0/4095442592 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8224107360 con 0x7f82240f5410 2024-02-19T22:11:21.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:21.787+0000 7f82217fa700 1 -- 172.21.15.88:0/4095442592 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f821c01c6c0 con 0x7f82240f5410 2024-02-19T22:11:21.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:21.787+0000 7f822aeed700 1 -- 172.21.15.88:0/4095442592 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8224045ad0 con 0x7f82240f5410 2024-02-19T22:11:21.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:21.787+0000 7f82217fa700 1 --2- 172.21.15.88:0/4095442592 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f820c02ff60 0x7f820c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:21.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:21.787+0000 7f82217fa700 1 -- 172.21.15.88:0/4095442592 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f821c041570 con 0x7f82240f5410 2024-02-19T22:11:21.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:21.787+0000 7f8223fff700 1 --2- 172.21.15.88:0/4095442592 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f820c02ff60 0x7f820c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:21.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:21.787+0000 7f8223fff700 1 --2- 172.21.15.88:0/4095442592 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f820c02ff60 0x7f820c032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f82180096a0 tx=0x7f8218006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:21.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:21.792+0000 7f82217fa700 1 -- 172.21.15.88:0/4095442592 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f821c012560 con 0x7f82240f5410 2024-02-19T22:11:21.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:21.998+0000 7f822aeed700 1 -- 172.21.15.88:0/4095442592 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8224108210 con 0x7f82240f5410 2024-02-19T22:11:22.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:21.999+0000 7f82217fa700 1 -- 172.21.15.88:0/4095442592 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f821c012380 con 0x7f82240f5410 2024-02-19T22:11:22.000 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:11:22.000 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:11:22.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:22.000+0000 7f822aeed700 1 -- 172.21.15.88:0/4095442592 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f820c02ff60 msgr2=0x7f820c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:22.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:22.000+0000 7f822aeed700 1 --2- 172.21.15.88:0/4095442592 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f820c02ff60 0x7f820c032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f82180096a0 tx=0x7f8218006b40).stop 2024-02-19T22:11:22.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:22.000+0000 7f822aeed700 1 -- 172.21.15.88:0/4095442592 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82240f5410 msgr2=0x7f82241067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:22.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:22.000+0000 7f822aeed700 1 --2- 172.21.15.88:0/4095442592 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82240f5410 0x7f82241067e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f821c004b70 tx=0x7f821c004d10).stop 2024-02-19T22:11:22.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:22.000+0000 7f822aeed700 1 -- 172.21.15.88:0/4095442592 shutdown_connections 2024-02-19T22:11:22.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:22.000+0000 7f822aeed700 1 --2- 172.21.15.88:0/4095442592 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f820c02ff60 0x7f820c032420 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:22.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:22.000+0000 7f822aeed700 1 --2- 172.21.15.88:0/4095442592 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82240f5410 0x7f82241067e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:22.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:22.000+0000 7f822aeed700 1 -- 172.21.15.88:0/4095442592 >> 172.21.15.88:0/4095442592 conn(0x7f82240f0b70 msgr2=0x7f82240f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:22.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:22.000+0000 7f822aeed700 1 -- 172.21.15.88:0/4095442592 shutdown_connections 2024-02-19T22:11:22.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:22.000+0000 7f822aeed700 1 -- 172.21.15.88:0/4095442592 wait complete. 2024-02-19T22:11:22.003 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:11:22.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:22 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:11:22.000034+0000 mon.smithi052 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.88:0/4095442592' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:11:23.562 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:11:23.562 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:11:23.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:23 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:11:21.627537+0000 mgr.smithi052.ghqrtk (mgr.14162) 50 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:25.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:25.363+0000 7fe794098700 1 -- 172.21.15.88:0/1245554141 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe78c0f5410 msgr2=0x7fe78c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:25.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:25.363+0000 7fe794098700 1 --2- 172.21.15.88:0/1245554141 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe78c0f5410 0x7fe78c0f5830 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fe77c009fb0 tx=0x7fe77c009b30).stop 2024-02-19T22:11:25.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:25.363+0000 7fe794098700 1 -- 172.21.15.88:0/1245554141 shutdown_connections 2024-02-19T22:11:25.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:25.363+0000 7fe794098700 1 --2- 172.21.15.88:0/1245554141 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe78c0f5410 0x7fe78c0f5830 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:25.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:25.363+0000 7fe794098700 1 -- 172.21.15.88:0/1245554141 >> 172.21.15.88:0/1245554141 conn(0x7fe78c0f0b70 msgr2=0x7fe78c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:25.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:25.363+0000 7fe794098700 1 -- 172.21.15.88:0/1245554141 shutdown_connections 2024-02-19T22:11:25.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:25.363+0000 7fe794098700 1 -- 172.21.15.88:0/1245554141 wait complete. 2024-02-19T22:11:25.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:25.364+0000 7fe794098700 1 Processor -- start 2024-02-19T22:11:25.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:25.364+0000 7fe794098700 1 -- start start 2024-02-19T22:11:25.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:25.364+0000 7fe794098700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe78c0f5410 0x7fe78c105720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:25.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:25.364+0000 7fe794098700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe78c105c10 con 0x7fe78c0f5410 2024-02-19T22:11:25.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:25.364+0000 7fe791e34700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe78c0f5410 0x7fe78c105720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:25.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:25.364+0000 7fe791e34700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe78c0f5410 0x7fe78c105720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:54412/0 (socket says 172.21.15.88:54412) 2024-02-19T22:11:25.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:25.364+0000 7fe791e34700 1 -- 172.21.15.88:0/3768504984 learned_addr learned my addr 172.21.15.88:0/3768504984 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:11:25.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:25.364+0000 7fe791e34700 1 -- 172.21.15.88:0/3768504984 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe77c005040 con 0x7fe78c0f5410 2024-02-19T22:11:25.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:25.365+0000 7fe791e34700 1 --2- 172.21.15.88:0/3768504984 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe78c0f5410 0x7fe78c105720 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fe77c005010 tx=0x7fe77c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:25.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:25.365+0000 7fe7827fc700 1 -- 172.21.15.88:0/3768504984 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe77c009320 con 0x7fe78c0f5410 2024-02-19T22:11:25.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:25.365+0000 7fe7827fc700 1 -- 172.21.15.88:0/3768504984 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe77c013380 con 0x7fe78c0f5410 2024-02-19T22:11:25.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:25.365+0000 7fe794098700 1 -- 172.21.15.88:0/3768504984 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe78c105e10 con 0x7fe78c0f5410 2024-02-19T22:11:25.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:25.365+0000 7fe7827fc700 1 -- 172.21.15.88:0/3768504984 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe77c014e50 con 0x7fe78c0f5410 2024-02-19T22:11:25.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:25.365+0000 7fe794098700 1 -- 172.21.15.88:0/3768504984 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe78c1062a0 con 0x7fe78c0f5410 2024-02-19T22:11:25.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:25.366+0000 7fe7827fc700 1 -- 172.21.15.88:0/3768504984 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe77c0095f0 con 0x7fe78c0f5410 2024-02-19T22:11:25.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:25.366+0000 7fe794098700 1 -- 172.21.15.88:0/3768504984 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe78c045ad0 con 0x7fe78c0f5410 2024-02-19T22:11:25.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:25.366+0000 7fe7827fc700 1 --2- 172.21.15.88:0/3768504984 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fe77802ff10 0x7fe7780323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:25.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:25.366+0000 7fe7827fc700 1 -- 172.21.15.88:0/3768504984 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe77c042d80 con 0x7fe78c0f5410 2024-02-19T22:11:25.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:25.366+0000 7fe791633700 1 --2- 172.21.15.88:0/3768504984 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fe77802ff10 0x7fe7780323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:25.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:25.367+0000 7fe791633700 1 --2- 172.21.15.88:0/3768504984 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fe77802ff10 0x7fe7780323d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fe7880096a0 tx=0x7fe788006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:25.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:25.371+0000 7fe7827fc700 1 -- 172.21.15.88:0/3768504984 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe77c014420 con 0x7fe78c0f5410 2024-02-19T22:11:25.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:25.574+0000 7fe794098700 1 -- 172.21.15.88:0/3768504984 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe78c107150 con 0x7fe78c0f5410 2024-02-19T22:11:25.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:25.574+0000 7fe7827fc700 1 -- 172.21.15.88:0/3768504984 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe77c011290 con 0x7fe78c0f5410 2024-02-19T22:11:25.576 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:11:25.576 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:11:25.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:25.576+0000 7fe794098700 1 -- 172.21.15.88:0/3768504984 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fe77802ff10 msgr2=0x7fe7780323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:25.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:25.576+0000 7fe794098700 1 --2- 172.21.15.88:0/3768504984 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fe77802ff10 0x7fe7780323d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fe7880096a0 tx=0x7fe788006b40).stop 2024-02-19T22:11:25.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:25.576+0000 7fe794098700 1 -- 172.21.15.88:0/3768504984 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe78c0f5410 msgr2=0x7fe78c105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:25.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:25.576+0000 7fe794098700 1 --2- 172.21.15.88:0/3768504984 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe78c0f5410 0x7fe78c105720 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fe77c005010 tx=0x7fe77c00b040).stop 2024-02-19T22:11:25.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:25.576+0000 7fe794098700 1 -- 172.21.15.88:0/3768504984 shutdown_connections 2024-02-19T22:11:25.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:25.576+0000 7fe794098700 1 --2- 172.21.15.88:0/3768504984 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fe77802ff10 0x7fe7780323d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:25.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:25.576+0000 7fe794098700 1 --2- 172.21.15.88:0/3768504984 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe78c0f5410 0x7fe78c105720 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:25.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:25.576+0000 7fe794098700 1 -- 172.21.15.88:0/3768504984 >> 172.21.15.88:0/3768504984 conn(0x7fe78c0f0b70 msgr2=0x7fe78c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:25.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:25.576+0000 7fe794098700 1 -- 172.21.15.88:0/3768504984 shutdown_connections 2024-02-19T22:11:25.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:25.576+0000 7fe794098700 1 -- 172.21.15.88:0/3768504984 wait complete. 2024-02-19T22:11:25.579 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:11:25.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:25 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:11:23.627982+0000 mgr.smithi052.ghqrtk (mgr.14162) 51 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:26.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:26 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:11:25.576016+0000 mon.smithi052 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.88:0/3768504984' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:11:27.184 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:11:27.185 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:11:27.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:27 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:11:25.628475+0000 mgr.smithi052.ghqrtk (mgr.14162) 52 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:28.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:28.967+0000 7fcd523b2700 1 -- 172.21.15.88:0/2894300926 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcd4c0f5410 msgr2=0x7fcd4c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:28.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:28.967+0000 7fcd523b2700 1 --2- 172.21.15.88:0/2894300926 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcd4c0f5410 0x7fcd4c0f5830 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fcd3c009fb0 tx=0x7fcd3c009b30).stop 2024-02-19T22:11:28.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:28.967+0000 7fcd523b2700 1 -- 172.21.15.88:0/2894300926 shutdown_connections 2024-02-19T22:11:28.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:28.967+0000 7fcd523b2700 1 --2- 172.21.15.88:0/2894300926 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcd4c0f5410 0x7fcd4c0f5830 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:28.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:28.967+0000 7fcd523b2700 1 -- 172.21.15.88:0/2894300926 >> 172.21.15.88:0/2894300926 conn(0x7fcd4c0f0b70 msgr2=0x7fcd4c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:28.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:28.968+0000 7fcd523b2700 1 -- 172.21.15.88:0/2894300926 shutdown_connections 2024-02-19T22:11:28.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:28.968+0000 7fcd523b2700 1 -- 172.21.15.88:0/2894300926 wait complete. 2024-02-19T22:11:28.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:28.968+0000 7fcd523b2700 1 Processor -- start 2024-02-19T22:11:28.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:28.968+0000 7fcd523b2700 1 -- start start 2024-02-19T22:11:28.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:28.969+0000 7fcd523b2700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcd4c0f5410 0x7fcd4c1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:28.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:28.969+0000 7fcd523b2700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd4c105bd0 con 0x7fcd4c0f5410 2024-02-19T22:11:28.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:28.969+0000 7fcd4bfff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcd4c0f5410 0x7fcd4c1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:28.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:28.969+0000 7fcd4bfff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcd4c0f5410 0x7fcd4c1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:54430/0 (socket says 172.21.15.88:54430) 2024-02-19T22:11:28.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:28.969+0000 7fcd4bfff700 1 -- 172.21.15.88:0/1497207856 learned_addr learned my addr 172.21.15.88:0/1497207856 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:11:28.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:28.969+0000 7fcd4bfff700 1 -- 172.21.15.88:0/1497207856 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd3c005040 con 0x7fcd4c0f5410 2024-02-19T22:11:28.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:28.969+0000 7fcd4bfff700 1 --2- 172.21.15.88:0/1497207856 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcd4c0f5410 0x7fcd4c1056e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fcd3c005010 tx=0x7fcd3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:28.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:28.969+0000 7fcd48ff9700 1 -- 172.21.15.88:0/1497207856 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd3c009320 con 0x7fcd4c0f5410 2024-02-19T22:11:28.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:28.969+0000 7fcd523b2700 1 -- 172.21.15.88:0/1497207856 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd4c105dd0 con 0x7fcd4c0f5410 2024-02-19T22:11:28.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:28.969+0000 7fcd48ff9700 1 -- 172.21.15.88:0/1497207856 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcd3c013380 con 0x7fcd4c0f5410 2024-02-19T22:11:28.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:28.969+0000 7fcd48ff9700 1 -- 172.21.15.88:0/1497207856 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd3c014e50 con 0x7fcd4c0f5410 2024-02-19T22:11:28.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:28.970+0000 7fcd523b2700 1 -- 172.21.15.88:0/1497207856 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd4c106260 con 0x7fcd4c0f5410 2024-02-19T22:11:28.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:28.970+0000 7fcd48ff9700 1 -- 172.21.15.88:0/1497207856 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcd3c0263a0 con 0x7fcd4c0f5410 2024-02-19T22:11:28.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:28.970+0000 7fcd48ff9700 1 --2- 172.21.15.88:0/1497207856 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fcd3402ff10 0x7fcd340323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:28.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:28.971+0000 7fcd48ff9700 1 -- 172.21.15.88:0/1497207856 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcd3c040f90 con 0x7fcd4c0f5410 2024-02-19T22:11:28.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:28.971+0000 7fcd523b2700 1 -- 172.21.15.88:0/1497207856 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd4c045ad0 con 0x7fcd4c0f5410 2024-02-19T22:11:28.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:28.971+0000 7fcd4b7fe700 1 --2- 172.21.15.88:0/1497207856 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fcd3402ff10 0x7fcd340323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:28.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:28.971+0000 7fcd4b7fe700 1 --2- 172.21.15.88:0/1497207856 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fcd3402ff10 0x7fcd340323d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7fcd400096a0 tx=0x7fcd40006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:28.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:28.975+0000 7fcd48ff9700 1 -- 172.21.15.88:0/1497207856 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcd3c012920 con 0x7fcd4c0f5410 2024-02-19T22:11:29.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:29.180+0000 7fcd523b2700 1 -- 172.21.15.88:0/1497207856 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcd4c106f80 con 0x7fcd4c0f5410 2024-02-19T22:11:29.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:29.181+0000 7fcd48ff9700 1 -- 172.21.15.88:0/1497207856 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcd3c02b020 con 0x7fcd4c0f5410 2024-02-19T22:11:29.182 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:11:29.182 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:11:29.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:29.182+0000 7fcd523b2700 1 -- 172.21.15.88:0/1497207856 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fcd3402ff10 msgr2=0x7fcd340323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:29.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:29.182+0000 7fcd523b2700 1 --2- 172.21.15.88:0/1497207856 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fcd3402ff10 0x7fcd340323d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7fcd400096a0 tx=0x7fcd40006b40).stop 2024-02-19T22:11:29.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:29.182+0000 7fcd523b2700 1 -- 172.21.15.88:0/1497207856 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcd4c0f5410 msgr2=0x7fcd4c1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:29.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:29.182+0000 7fcd523b2700 1 --2- 172.21.15.88:0/1497207856 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcd4c0f5410 0x7fcd4c1056e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fcd3c005010 tx=0x7fcd3c00b040).stop 2024-02-19T22:11:29.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:29.182+0000 7fcd523b2700 1 -- 172.21.15.88:0/1497207856 shutdown_connections 2024-02-19T22:11:29.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:29.182+0000 7fcd523b2700 1 --2- 172.21.15.88:0/1497207856 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fcd3402ff10 0x7fcd340323d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:29.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:29.182+0000 7fcd523b2700 1 --2- 172.21.15.88:0/1497207856 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcd4c0f5410 0x7fcd4c1056e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:29.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:29.182+0000 7fcd523b2700 1 -- 172.21.15.88:0/1497207856 >> 172.21.15.88:0/1497207856 conn(0x7fcd4c0f0b70 msgr2=0x7fcd4c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:29.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:29.182+0000 7fcd523b2700 1 -- 172.21.15.88:0/1497207856 shutdown_connections 2024-02-19T22:11:29.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:29.182+0000 7fcd523b2700 1 -- 172.21.15.88:0/1497207856 wait complete. 2024-02-19T22:11:29.185 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:11:29.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:29 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:11:27.628928+0000 mgr.smithi052.ghqrtk (mgr.14162) 53 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:30.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:30 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:11:29.182209+0000 mon.smithi052 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.88:0/1497207856' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:11:30.826 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:11:30.827 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:11:31.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:31 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:11:29.629419+0000 mgr.smithi052.ghqrtk (mgr.14162) 54 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:32.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:32.630+0000 7f2bea319700 1 -- 172.21.15.88:0/3735966354 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2be4060280 msgr2=0x7f2be40f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:32.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:32.630+0000 7f2bea319700 1 --2- 172.21.15.88:0/3735966354 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2be4060280 0x7f2be40f90f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f2bd4009fb0 tx=0x7f2bd4009b30).stop 2024-02-19T22:11:32.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:32.630+0000 7f2bea319700 1 -- 172.21.15.88:0/3735966354 shutdown_connections 2024-02-19T22:11:32.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:32.630+0000 7f2bea319700 1 --2- 172.21.15.88:0/3735966354 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2be4060280 0x7f2be40f90f0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:32.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:32.630+0000 7f2bea319700 1 -- 172.21.15.88:0/3735966354 >> 172.21.15.88:0/3735966354 conn(0x7f2be40f0b70 msgr2=0x7f2be40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:32.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:32.630+0000 7f2bea319700 1 -- 172.21.15.88:0/3735966354 shutdown_connections 2024-02-19T22:11:32.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:32.630+0000 7f2bea319700 1 -- 172.21.15.88:0/3735966354 wait complete. 2024-02-19T22:11:32.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:32.631+0000 7f2bea319700 1 Processor -- start 2024-02-19T22:11:32.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:32.631+0000 7f2bea319700 1 -- start start 2024-02-19T22:11:32.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:32.631+0000 7f2bea319700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2be4060280 0x7f2be4106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:32.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:32.631+0000 7f2bea319700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be4106c70 con 0x7f2be4060280 2024-02-19T22:11:32.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:32.631+0000 7f2be3fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2be4060280 0x7f2be4106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:32.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:32.631+0000 7f2be3fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2be4060280 0x7f2be4106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:54448/0 (socket says 172.21.15.88:54448) 2024-02-19T22:11:32.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:32.631+0000 7f2be3fff700 1 -- 172.21.15.88:0/2818175050 learned_addr learned my addr 172.21.15.88:0/2818175050 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:11:32.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:32.632+0000 7f2be3fff700 1 -- 172.21.15.88:0/2818175050 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bd4005040 con 0x7f2be4060280 2024-02-19T22:11:32.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:32.632+0000 7f2be3fff700 1 --2- 172.21.15.88:0/2818175050 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2be4060280 0x7f2be4106780 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f2bd40089f0 tx=0x7f2bd400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:32.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:32.632+0000 7f2be0ff9700 1 -- 172.21.15.88:0/2818175050 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bd4009320 con 0x7f2be4060280 2024-02-19T22:11:32.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:32.632+0000 7f2bea319700 1 -- 172.21.15.88:0/2818175050 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2be4106e70 con 0x7f2be4060280 2024-02-19T22:11:32.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:32.632+0000 7f2be0ff9700 1 -- 172.21.15.88:0/2818175050 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2bd4013380 con 0x7f2be4060280 2024-02-19T22:11:32.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:32.632+0000 7f2be0ff9700 1 -- 172.21.15.88:0/2818175050 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bd4014e20 con 0x7f2be4060280 2024-02-19T22:11:32.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:32.632+0000 7f2bea319700 1 -- 172.21.15.88:0/2818175050 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2be4107300 con 0x7f2be4060280 2024-02-19T22:11:32.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:32.633+0000 7f2be0ff9700 1 -- 172.21.15.88:0/2818175050 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2bd40095f0 con 0x7f2be4060280 2024-02-19T22:11:32.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:32.633+0000 7f2bea319700 1 -- 172.21.15.88:0/2818175050 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2be4045ad0 con 0x7f2be4060280 2024-02-19T22:11:32.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:32.633+0000 7f2be0ff9700 1 --2- 172.21.15.88:0/2818175050 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f2bcc02ff10 0x7f2bcc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:32.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:32.633+0000 7f2be0ff9700 1 -- 172.21.15.88:0/2818175050 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2bd4040f10 con 0x7f2be4060280 2024-02-19T22:11:32.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:32.633+0000 7f2be37fe700 1 --2- 172.21.15.88:0/2818175050 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f2bcc02ff10 0x7f2bcc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:32.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:32.634+0000 7f2be37fe700 1 --2- 172.21.15.88:0/2818175050 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f2bcc02ff10 0x7f2bcc0323d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f2bd80096a0 tx=0x7f2bd8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:32.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:32.638+0000 7f2be0ff9700 1 -- 172.21.15.88:0/2818175050 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2bd4014420 con 0x7f2be4060280 2024-02-19T22:11:32.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:32.844+0000 7f2bea319700 1 -- 172.21.15.88:0/2818175050 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2be41081b0 con 0x7f2be4060280 2024-02-19T22:11:32.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:32.845+0000 7f2be0ff9700 1 -- 172.21.15.88:0/2818175050 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2bd401d9d0 con 0x7f2be4060280 2024-02-19T22:11:32.845 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:11:32.845 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:11:32.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:32.846+0000 7f2bea319700 1 -- 172.21.15.88:0/2818175050 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f2bcc02ff10 msgr2=0x7f2bcc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:32.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:32.846+0000 7f2bea319700 1 --2- 172.21.15.88:0/2818175050 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f2bcc02ff10 0x7f2bcc0323d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f2bd80096a0 tx=0x7f2bd8006b40).stop 2024-02-19T22:11:32.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:32.846+0000 7f2bea319700 1 -- 172.21.15.88:0/2818175050 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2be4060280 msgr2=0x7f2be4106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:32.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:32.846+0000 7f2bea319700 1 --2- 172.21.15.88:0/2818175050 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2be4060280 0x7f2be4106780 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f2bd40089f0 tx=0x7f2bd400b040).stop 2024-02-19T22:11:32.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:32.846+0000 7f2bea319700 1 -- 172.21.15.88:0/2818175050 shutdown_connections 2024-02-19T22:11:32.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:32.846+0000 7f2bea319700 1 --2- 172.21.15.88:0/2818175050 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f2bcc02ff10 0x7f2bcc0323d0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:32.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:32.846+0000 7f2bea319700 1 --2- 172.21.15.88:0/2818175050 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2be4060280 0x7f2be4106780 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:32.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:32.846+0000 7f2bea319700 1 -- 172.21.15.88:0/2818175050 >> 172.21.15.88:0/2818175050 conn(0x7f2be40f0b70 msgr2=0x7f2be40f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:32.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:32.846+0000 7f2bea319700 1 -- 172.21.15.88:0/2818175050 shutdown_connections 2024-02-19T22:11:32.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:32.846+0000 7f2bea319700 1 -- 172.21.15.88:0/2818175050 wait complete. 2024-02-19T22:11:32.848 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:11:33.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:33 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:11:31.629881+0000 mgr.smithi052.ghqrtk (mgr.14162) 55 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:33.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:33 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:11:32.845013+0000 mon.smithi052 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.88:0/2818175050' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:11:34.471 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:11:34.471 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:11:35.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:35 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:11:33.630291+0000 mgr.smithi052.ghqrtk (mgr.14162) 56 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:36.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:36.439+0000 7f7a78259700 1 -- 172.21.15.88:0/1995436938 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7a700f6510 msgr2=0x7f7a700f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:36.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:36.439+0000 7f7a78259700 1 --2- 172.21.15.88:0/1995436938 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7a700f6510 0x7f7a700f6930 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f7a64009fb0 tx=0x7f7a64009b30).stop 2024-02-19T22:11:36.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:36.439+0000 7f7a78259700 1 -- 172.21.15.88:0/1995436938 shutdown_connections 2024-02-19T22:11:36.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:36.439+0000 7f7a78259700 1 --2- 172.21.15.88:0/1995436938 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7a700f6510 0x7f7a700f6930 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:36.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:36.439+0000 7f7a78259700 1 -- 172.21.15.88:0/1995436938 >> 172.21.15.88:0/1995436938 conn(0x7f7a700f1c70 msgr2=0x7f7a700f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:36.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:36.439+0000 7f7a78259700 1 -- 172.21.15.88:0/1995436938 shutdown_connections 2024-02-19T22:11:36.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:36.439+0000 7f7a78259700 1 -- 172.21.15.88:0/1995436938 wait complete. 2024-02-19T22:11:36.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:36.439+0000 7f7a78259700 1 Processor -- start 2024-02-19T22:11:36.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:36.440+0000 7f7a78259700 1 -- start start 2024-02-19T22:11:36.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:36.440+0000 7f7a78259700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7a700f6510 0x7f7a701067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:36.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:36.440+0000 7f7a78259700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a70106cd0 con 0x7f7a700f6510 2024-02-19T22:11:36.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:36.440+0000 7f7a75ff5700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7a700f6510 0x7f7a701067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:36.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:36.440+0000 7f7a75ff5700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7a700f6510 0x7f7a701067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:47966/0 (socket says 172.21.15.88:47966) 2024-02-19T22:11:36.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:36.440+0000 7f7a75ff5700 1 -- 172.21.15.88:0/3351761602 learned_addr learned my addr 172.21.15.88:0/3351761602 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:11:36.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:36.440+0000 7f7a75ff5700 1 -- 172.21.15.88:0/3351761602 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a64005040 con 0x7f7a700f6510 2024-02-19T22:11:36.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:36.440+0000 7f7a75ff5700 1 --2- 172.21.15.88:0/3351761602 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7a700f6510 0x7f7a701067e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f7a64005010 tx=0x7f7a6400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:36.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:36.440+0000 7f7a627fc700 1 -- 172.21.15.88:0/3351761602 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a64009320 con 0x7f7a700f6510 2024-02-19T22:11:36.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:36.440+0000 7f7a78259700 1 -- 172.21.15.88:0/3351761602 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a70106ed0 con 0x7f7a700f6510 2024-02-19T22:11:36.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:36.441+0000 7f7a627fc700 1 -- 172.21.15.88:0/3351761602 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7a64013380 con 0x7f7a700f6510 2024-02-19T22:11:36.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:36.441+0000 7f7a627fc700 1 -- 172.21.15.88:0/3351761602 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a64014e50 con 0x7f7a700f6510 2024-02-19T22:11:36.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:36.441+0000 7f7a78259700 1 -- 172.21.15.88:0/3351761602 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a70107360 con 0x7f7a700f6510 2024-02-19T22:11:36.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:36.441+0000 7f7a627fc700 1 -- 172.21.15.88:0/3351761602 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7a640263a0 con 0x7f7a700f6510 2024-02-19T22:11:36.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:36.441+0000 7f7a627fc700 1 --2- 172.21.15.88:0/3351761602 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f7a5c02ff10 0x7f7a5c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:36.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:36.441+0000 7f7a78259700 1 -- 172.21.15.88:0/3351761602 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a70045ad0 con 0x7f7a700f6510 2024-02-19T22:11:36.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:36.442+0000 7f7a627fc700 1 -- 172.21.15.88:0/3351761602 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7a64041290 con 0x7f7a700f6510 2024-02-19T22:11:36.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:36.442+0000 7f7a757f4700 1 --2- 172.21.15.88:0/3351761602 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f7a5c02ff10 0x7f7a5c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:36.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:36.442+0000 7f7a757f4700 1 --2- 172.21.15.88:0/3351761602 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f7a5c02ff10 0x7f7a5c0323d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f7a6c0096a0 tx=0x7f7a6c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:36.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:36.446+0000 7f7a627fc700 1 -- 172.21.15.88:0/3351761602 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7a64012920 con 0x7f7a700f6510 2024-02-19T22:11:36.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:36.651+0000 7f7a78259700 1 -- 172.21.15.88:0/3351761602 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7a701081b0 con 0x7f7a700f6510 2024-02-19T22:11:36.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:36.652+0000 7f7a627fc700 1 -- 172.21.15.88:0/3351761602 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7a64012330 con 0x7f7a700f6510 2024-02-19T22:11:36.652 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:11:36.652 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:11:36.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:36.653+0000 7f7a78259700 1 -- 172.21.15.88:0/3351761602 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f7a5c02ff10 msgr2=0x7f7a5c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:36.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:36.653+0000 7f7a78259700 1 --2- 172.21.15.88:0/3351761602 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f7a5c02ff10 0x7f7a5c0323d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f7a6c0096a0 tx=0x7f7a6c006b40).stop 2024-02-19T22:11:36.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:36.653+0000 7f7a78259700 1 -- 172.21.15.88:0/3351761602 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7a700f6510 msgr2=0x7f7a701067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:36.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:36.653+0000 7f7a78259700 1 --2- 172.21.15.88:0/3351761602 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7a700f6510 0x7f7a701067e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f7a64005010 tx=0x7f7a6400b040).stop 2024-02-19T22:11:36.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:36.653+0000 7f7a78259700 1 -- 172.21.15.88:0/3351761602 shutdown_connections 2024-02-19T22:11:36.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:36.653+0000 7f7a78259700 1 --2- 172.21.15.88:0/3351761602 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f7a5c02ff10 0x7f7a5c0323d0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:36.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:36.653+0000 7f7a78259700 1 --2- 172.21.15.88:0/3351761602 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7a700f6510 0x7f7a701067e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:36.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:36.653+0000 7f7a78259700 1 -- 172.21.15.88:0/3351761602 >> 172.21.15.88:0/3351761602 conn(0x7f7a700f1c70 msgr2=0x7f7a700f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:36.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:36.653+0000 7f7a78259700 1 -- 172.21.15.88:0/3351761602 shutdown_connections 2024-02-19T22:11:36.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:36.653+0000 7f7a78259700 1 -- 172.21.15.88:0/3351761602 wait complete. 2024-02-19T22:11:36.655 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:11:37.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:37 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:11:35.630602+0000 mgr.smithi052.ghqrtk (mgr.14162) 57 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:37.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:37 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:11:36.652151+0000 mon.smithi052 (mon.0) 139 : audit [DBG] from='client.? 172.21.15.88:0/3351761602' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:11:38.288 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:11:38.288 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:11:39.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:11:37.631004+0000 mgr.smithi052.ghqrtk (mgr.14162) 58 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:40.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:40.079+0000 7fcac80d6700 1 -- 172.21.15.88:0/1772997408 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcac00f6510 msgr2=0x7fcac00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:40.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:40.079+0000 7fcac80d6700 1 --2- 172.21.15.88:0/1772997408 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcac00f6510 0x7fcac00f6930 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fcabc009fb0 tx=0x7fcabc009b30).stop 2024-02-19T22:11:40.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:40.079+0000 7fcac80d6700 1 -- 172.21.15.88:0/1772997408 shutdown_connections 2024-02-19T22:11:40.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:40.079+0000 7fcac80d6700 1 --2- 172.21.15.88:0/1772997408 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcac00f6510 0x7fcac00f6930 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:40.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:40.079+0000 7fcac80d6700 1 -- 172.21.15.88:0/1772997408 >> 172.21.15.88:0/1772997408 conn(0x7fcac00f1c70 msgr2=0x7fcac00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:40.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:40.079+0000 7fcac80d6700 1 -- 172.21.15.88:0/1772997408 shutdown_connections 2024-02-19T22:11:40.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:40.079+0000 7fcac80d6700 1 -- 172.21.15.88:0/1772997408 wait complete. 2024-02-19T22:11:40.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:40.079+0000 7fcac80d6700 1 Processor -- start 2024-02-19T22:11:40.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:40.079+0000 7fcac80d6700 1 -- start start 2024-02-19T22:11:40.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:40.080+0000 7fcac80d6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcac00f6510 0x7fcac006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:40.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:40.080+0000 7fcac80d6700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac0069e20 con 0x7fcac00f6510 2024-02-19T22:11:40.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:40.080+0000 7fcac5e72700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcac00f6510 0x7fcac006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:40.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:40.080+0000 7fcac5e72700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcac00f6510 0x7fcac006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:47986/0 (socket says 172.21.15.88:47986) 2024-02-19T22:11:40.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:40.080+0000 7fcac5e72700 1 -- 172.21.15.88:0/3030400595 learned_addr learned my addr 172.21.15.88:0/3030400595 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:11:40.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:40.080+0000 7fcac5e72700 1 -- 172.21.15.88:0/3030400595 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcabc005040 con 0x7fcac00f6510 2024-02-19T22:11:40.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:40.080+0000 7fcac5e72700 1 --2- 172.21.15.88:0/3030400595 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcac00f6510 0x7fcac006b9a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fcabc005010 tx=0x7fcabc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:40.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:40.081+0000 7fcab67fc700 1 -- 172.21.15.88:0/3030400595 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcabc009320 con 0x7fcac00f6510 2024-02-19T22:11:40.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:40.081+0000 7fcac80d6700 1 -- 172.21.15.88:0/3030400595 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcac0069fc0 con 0x7fcac00f6510 2024-02-19T22:11:40.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:40.081+0000 7fcab67fc700 1 -- 172.21.15.88:0/3030400595 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcabc013380 con 0x7fcac00f6510 2024-02-19T22:11:40.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:40.081+0000 7fcab67fc700 1 -- 172.21.15.88:0/3030400595 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcabc014e50 con 0x7fcac00f6510 2024-02-19T22:11:40.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:40.081+0000 7fcac80d6700 1 -- 172.21.15.88:0/3030400595 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcac006a450 con 0x7fcac00f6510 2024-02-19T22:11:40.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:40.081+0000 7fcab67fc700 1 -- 172.21.15.88:0/3030400595 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcabc0095f0 con 0x7fcac00f6510 2024-02-19T22:11:40.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:40.081+0000 7fcac80d6700 1 -- 172.21.15.88:0/3030400595 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcac0059100 con 0x7fcac00f6510 2024-02-19T22:11:40.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:40.082+0000 7fcab67fc700 1 --2- 172.21.15.88:0/3030400595 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fcaac02ff10 0x7fcaac0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:40.084 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:40.082+0000 7fcab67fc700 1 -- 172.21.15.88:0/3030400595 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcabc040fa0 con 0x7fcac00f6510 2024-02-19T22:11:40.084 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:40.082+0000 7fcac5671700 1 --2- 172.21.15.88:0/3030400595 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fcaac02ff10 0x7fcaac0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:40.084 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:40.082+0000 7fcac5671700 1 --2- 172.21.15.88:0/3030400595 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fcaac02ff10 0x7fcaac0323d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fcab00096a0 tx=0x7fcab0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:40.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:40.086+0000 7fcab67fc700 1 -- 172.21.15.88:0/3030400595 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcabc014420 con 0x7fcac00f6510 2024-02-19T22:11:40.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:40.291+0000 7fcac80d6700 1 -- 172.21.15.88:0/3030400595 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcac00fc0a0 con 0x7fcac00f6510 2024-02-19T22:11:40.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:40.292+0000 7fcab67fc700 1 -- 172.21.15.88:0/3030400595 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcabc010230 con 0x7fcac00f6510 2024-02-19T22:11:40.293 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:11:40.293 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:11:40.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:40.294+0000 7fcac80d6700 1 -- 172.21.15.88:0/3030400595 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fcaac02ff10 msgr2=0x7fcaac0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:40.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:40.294+0000 7fcac80d6700 1 --2- 172.21.15.88:0/3030400595 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fcaac02ff10 0x7fcaac0323d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fcab00096a0 tx=0x7fcab0006b40).stop 2024-02-19T22:11:40.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:40.294+0000 7fcac80d6700 1 -- 172.21.15.88:0/3030400595 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcac00f6510 msgr2=0x7fcac006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:40.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:40.294+0000 7fcac80d6700 1 --2- 172.21.15.88:0/3030400595 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcac00f6510 0x7fcac006b9a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fcabc005010 tx=0x7fcabc00b040).stop 2024-02-19T22:11:40.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:40.294+0000 7fcac80d6700 1 -- 172.21.15.88:0/3030400595 shutdown_connections 2024-02-19T22:11:40.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:40.294+0000 7fcac80d6700 1 --2- 172.21.15.88:0/3030400595 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fcaac02ff10 0x7fcaac0323d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:40.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:40.294+0000 7fcac80d6700 1 --2- 172.21.15.88:0/3030400595 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcac00f6510 0x7fcac006b9a0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:40.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:40.294+0000 7fcac80d6700 1 -- 172.21.15.88:0/3030400595 >> 172.21.15.88:0/3030400595 conn(0x7fcac00f1c70 msgr2=0x7fcac00f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:40.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:40.294+0000 7fcac80d6700 1 -- 172.21.15.88:0/3030400595 shutdown_connections 2024-02-19T22:11:40.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:40.294+0000 7fcac80d6700 1 -- 172.21.15.88:0/3030400595 wait complete. 2024-02-19T22:11:40.295 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:11:41.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:41 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:11:39.631349+0000 mgr.smithi052.ghqrtk (mgr.14162) 59 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:41.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:41 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:11:40.292473+0000 mon.smithi052 (mon.0) 140 : audit [DBG] from='client.? 172.21.15.88:0/3030400595' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:11:41.883 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:11:41.883 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:11:43.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:43 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:11:41.631698+0000 mgr.smithi052.ghqrtk (mgr.14162) 60 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:43.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:43.729+0000 7f25495c8700 1 -- 172.21.15.88:0/370434457 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f25440f5410 msgr2=0x7f25440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:43.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:43.729+0000 7f25495c8700 1 --2- 172.21.15.88:0/370434457 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f25440f5410 0x7f25440f5830 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f2538009fb0 tx=0x7f2538009b30).stop 2024-02-19T22:11:43.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:43.729+0000 7f25495c8700 1 -- 172.21.15.88:0/370434457 shutdown_connections 2024-02-19T22:11:43.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:43.729+0000 7f25495c8700 1 --2- 172.21.15.88:0/370434457 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f25440f5410 0x7f25440f5830 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:43.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:43.729+0000 7f25495c8700 1 -- 172.21.15.88:0/370434457 >> 172.21.15.88:0/370434457 conn(0x7f25440f0b70 msgr2=0x7f25440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:43.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:43.729+0000 7f25495c8700 1 -- 172.21.15.88:0/370434457 shutdown_connections 2024-02-19T22:11:43.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:43.729+0000 7f25495c8700 1 -- 172.21.15.88:0/370434457 wait complete. 2024-02-19T22:11:43.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:43.730+0000 7f25495c8700 1 Processor -- start 2024-02-19T22:11:43.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:43.730+0000 7f25495c8700 1 -- start start 2024-02-19T22:11:43.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:43.730+0000 7f25495c8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f25440f5410 0x7f2544106810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:43.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:43.730+0000 7f25495c8700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2544106d00 con 0x7f25440f5410 2024-02-19T22:11:43.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:43.730+0000 7f2542ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f25440f5410 0x7f2544106810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:43.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:43.730+0000 7f2542ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f25440f5410 0x7f2544106810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:33066/0 (socket says 172.21.15.88:33066) 2024-02-19T22:11:43.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:43.730+0000 7f2542ffd700 1 -- 172.21.15.88:0/882096083 learned_addr learned my addr 172.21.15.88:0/882096083 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:11:43.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:43.730+0000 7f2542ffd700 1 -- 172.21.15.88:0/882096083 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2538005040 con 0x7f25440f5410 2024-02-19T22:11:43.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:43.731+0000 7f2542ffd700 1 --2- 172.21.15.88:0/882096083 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f25440f5410 0x7f2544106810 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f2538005010 tx=0x7f253800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:43.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:43.731+0000 7f252b7fe700 1 -- 172.21.15.88:0/882096083 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2538009320 con 0x7f25440f5410 2024-02-19T22:11:43.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:43.731+0000 7f25495c8700 1 -- 172.21.15.88:0/882096083 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2544106f00 con 0x7f25440f5410 2024-02-19T22:11:43.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:43.731+0000 7f252b7fe700 1 -- 172.21.15.88:0/882096083 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2538013380 con 0x7f25440f5410 2024-02-19T22:11:43.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:43.731+0000 7f252b7fe700 1 -- 172.21.15.88:0/882096083 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2538014e50 con 0x7f25440f5410 2024-02-19T22:11:43.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:43.731+0000 7f25495c8700 1 -- 172.21.15.88:0/882096083 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2544107390 con 0x7f25440f5410 2024-02-19T22:11:43.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:43.732+0000 7f252b7fe700 1 -- 172.21.15.88:0/882096083 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f25380095f0 con 0x7f25440f5410 2024-02-19T22:11:43.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:43.732+0000 7f25495c8700 1 -- 172.21.15.88:0/882096083 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2544045ad0 con 0x7f25440f5410 2024-02-19T22:11:43.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:43.732+0000 7f252b7fe700 1 --2- 172.21.15.88:0/882096083 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f252c02ff10 0x7f252c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:43.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:43.732+0000 7f252b7fe700 1 -- 172.21.15.88:0/882096083 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2538040f10 con 0x7f25440f5410 2024-02-19T22:11:43.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:43.732+0000 7f25427fc700 1 --2- 172.21.15.88:0/882096083 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f252c02ff10 0x7f252c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:43.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:43.732+0000 7f25427fc700 1 --2- 172.21.15.88:0/882096083 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f252c02ff10 0x7f252c0323d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f25340096a0 tx=0x7f2534006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:43.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:43.736+0000 7f252b7fe700 1 -- 172.21.15.88:0/882096083 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2538014420 con 0x7f25440f5410 2024-02-19T22:11:43.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:43.938+0000 7f25495c8700 1 -- 172.21.15.88:0/882096083 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f25440fafa0 con 0x7f25440f5410 2024-02-19T22:11:43.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:43.939+0000 7f252b7fe700 1 -- 172.21.15.88:0/882096083 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2538010230 con 0x7f25440f5410 2024-02-19T22:11:43.940 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:11:43.940 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:11:43.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:43.941+0000 7f25495c8700 1 -- 172.21.15.88:0/882096083 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f252c02ff10 msgr2=0x7f252c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:43.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:43.941+0000 7f25495c8700 1 --2- 172.21.15.88:0/882096083 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f252c02ff10 0x7f252c0323d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f25340096a0 tx=0x7f2534006b40).stop 2024-02-19T22:11:43.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:43.941+0000 7f25495c8700 1 -- 172.21.15.88:0/882096083 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f25440f5410 msgr2=0x7f2544106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:43.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:43.941+0000 7f25495c8700 1 --2- 172.21.15.88:0/882096083 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f25440f5410 0x7f2544106810 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f2538005010 tx=0x7f253800b040).stop 2024-02-19T22:11:43.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:43.941+0000 7f25495c8700 1 -- 172.21.15.88:0/882096083 shutdown_connections 2024-02-19T22:11:43.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:43.941+0000 7f25495c8700 1 --2- 172.21.15.88:0/882096083 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f252c02ff10 0x7f252c0323d0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:43.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:43.941+0000 7f25495c8700 1 --2- 172.21.15.88:0/882096083 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f25440f5410 0x7f2544106810 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:43.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:43.941+0000 7f25495c8700 1 -- 172.21.15.88:0/882096083 >> 172.21.15.88:0/882096083 conn(0x7f25440f0b70 msgr2=0x7f254405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:43.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:43.941+0000 7f25495c8700 1 -- 172.21.15.88:0/882096083 shutdown_connections 2024-02-19T22:11:43.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:43.941+0000 7f25495c8700 1 -- 172.21.15.88:0/882096083 wait complete. 2024-02-19T22:11:43.942 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:11:44.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:44 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:11:43.746157+0000 mon.smithi052 (mon.0) 141 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi052.ghqrtk/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:11:44.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:44 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:11:43.749688+0000 mon.smithi052 (mon.0) 142 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi052.ghqrtk/trash_purge_schedule"}]: dispatch 2024-02-19T22:11:44.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:44 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:11:43.939611+0000 mon.smithi052 (mon.0) 143 : audit [DBG] from='client.? 172.21.15.88:0/882096083' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:11:45.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:45 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:11:43.632144+0000 mgr.smithi052.ghqrtk (mgr.14162) 61 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:45.640 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:11:45.640 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:11:47.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:47.490+0000 7fdea7871700 1 -- 172.21.15.88:0/771077828 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdea00f5410 msgr2=0x7fdea00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:47.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:47.490+0000 7fdea7871700 1 --2- 172.21.15.88:0/771077828 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdea00f5410 0x7fdea00f5830 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fde94009fb0 tx=0x7fde94009b30).stop 2024-02-19T22:11:47.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:47.490+0000 7fdea7871700 1 -- 172.21.15.88:0/771077828 shutdown_connections 2024-02-19T22:11:47.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:47.490+0000 7fdea7871700 1 --2- 172.21.15.88:0/771077828 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdea00f5410 0x7fdea00f5830 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:47.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:47.490+0000 7fdea7871700 1 -- 172.21.15.88:0/771077828 >> 172.21.15.88:0/771077828 conn(0x7fdea00f0b70 msgr2=0x7fdea00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:47.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:47.490+0000 7fdea7871700 1 -- 172.21.15.88:0/771077828 shutdown_connections 2024-02-19T22:11:47.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:47.490+0000 7fdea7871700 1 -- 172.21.15.88:0/771077828 wait complete. 2024-02-19T22:11:47.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:47.491+0000 7fdea7871700 1 Processor -- start 2024-02-19T22:11:47.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:47.491+0000 7fdea7871700 1 -- start start 2024-02-19T22:11:47.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:47.491+0000 7fdea7871700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdea00f5410 0x7fdea01056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:47.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:47.491+0000 7fdea7871700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdea0105bd0 con 0x7fdea00f5410 2024-02-19T22:11:47.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:47.491+0000 7fdea560d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdea00f5410 0x7fdea01056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:47.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:47.491+0000 7fdea560d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdea00f5410 0x7fdea01056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:33082/0 (socket says 172.21.15.88:33082) 2024-02-19T22:11:47.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:47.491+0000 7fdea560d700 1 -- 172.21.15.88:0/326794978 learned_addr learned my addr 172.21.15.88:0/326794978 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:11:47.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:47.491+0000 7fdea560d700 1 -- 172.21.15.88:0/326794978 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde94005040 con 0x7fdea00f5410 2024-02-19T22:11:47.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:47.492+0000 7fdea560d700 1 --2- 172.21.15.88:0/326794978 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdea00f5410 0x7fdea01056e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fde94005010 tx=0x7fde9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:47.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:47.492+0000 7fde91ffb700 1 -- 172.21.15.88:0/326794978 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde94009320 con 0x7fdea00f5410 2024-02-19T22:11:47.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:47.492+0000 7fdea7871700 1 -- 172.21.15.88:0/326794978 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdea0105dd0 con 0x7fdea00f5410 2024-02-19T22:11:47.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:47.492+0000 7fde91ffb700 1 -- 172.21.15.88:0/326794978 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fde94013380 con 0x7fdea00f5410 2024-02-19T22:11:47.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:47.492+0000 7fde91ffb700 1 -- 172.21.15.88:0/326794978 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde94014e50 con 0x7fdea00f5410 2024-02-19T22:11:47.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:47.492+0000 7fdea7871700 1 -- 172.21.15.88:0/326794978 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdea0106260 con 0x7fdea00f5410 2024-02-19T22:11:47.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:47.493+0000 7fde91ffb700 1 -- 172.21.15.88:0/326794978 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fde940095f0 con 0x7fdea00f5410 2024-02-19T22:11:47.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:47.493+0000 7fdea7871700 1 -- 172.21.15.88:0/326794978 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdea0045ad0 con 0x7fdea00f5410 2024-02-19T22:11:47.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:47.493+0000 7fde91ffb700 1 --2- 172.21.15.88:0/326794978 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fde8c02ff10 0x7fde8c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:47.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:47.493+0000 7fde91ffb700 1 -- 172.21.15.88:0/326794978 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fde94040e60 con 0x7fdea00f5410 2024-02-19T22:11:47.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:47.493+0000 7fdea4e0c700 1 --2- 172.21.15.88:0/326794978 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fde8c02ff10 0x7fde8c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:47.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:47.494+0000 7fdea4e0c700 1 --2- 172.21.15.88:0/326794978 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fde8c02ff10 0x7fde8c0323d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fde9c0096a0 tx=0x7fde9c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:47.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:47.498+0000 7fde91ffb700 1 -- 172.21.15.88:0/326794978 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fde94014420 con 0x7fdea00f5410 2024-02-19T22:11:47.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:47 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:11:45.632460+0000 mgr.smithi052.ghqrtk (mgr.14162) 62 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:47.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:47.701+0000 7fdea7871700 1 -- 172.21.15.88:0/326794978 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdea0107110 con 0x7fdea00f5410 2024-02-19T22:11:47.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:47.702+0000 7fde91ffb700 1 -- 172.21.15.88:0/326794978 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fde9401d980 con 0x7fdea00f5410 2024-02-19T22:11:47.704 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:11:47.704 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:11:47.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:47.704+0000 7fdea7871700 1 -- 172.21.15.88:0/326794978 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fde8c02ff10 msgr2=0x7fde8c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:47.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:47.704+0000 7fdea7871700 1 --2- 172.21.15.88:0/326794978 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fde8c02ff10 0x7fde8c0323d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fde9c0096a0 tx=0x7fde9c006b40).stop 2024-02-19T22:11:47.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:47.704+0000 7fdea7871700 1 -- 172.21.15.88:0/326794978 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdea00f5410 msgr2=0x7fdea01056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:47.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:47.704+0000 7fdea7871700 1 --2- 172.21.15.88:0/326794978 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdea00f5410 0x7fdea01056e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fde94005010 tx=0x7fde9400b040).stop 2024-02-19T22:11:47.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:47.704+0000 7fdea7871700 1 -- 172.21.15.88:0/326794978 shutdown_connections 2024-02-19T22:11:47.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:47.704+0000 7fdea7871700 1 --2- 172.21.15.88:0/326794978 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fde8c02ff10 0x7fde8c0323d0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:47.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:47.704+0000 7fdea7871700 1 --2- 172.21.15.88:0/326794978 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdea00f5410 0x7fdea01056e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:47.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:47.704+0000 7fdea7871700 1 -- 172.21.15.88:0/326794978 >> 172.21.15.88:0/326794978 conn(0x7fdea00f0b70 msgr2=0x7fdea00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:47.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:47.704+0000 7fdea7871700 1 -- 172.21.15.88:0/326794978 shutdown_connections 2024-02-19T22:11:47.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:47.704+0000 7fdea7871700 1 -- 172.21.15.88:0/326794978 wait complete. 2024-02-19T22:11:47.706 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:11:48.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:48 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:11:47.703525+0000 mon.smithi052 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.88:0/326794978' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:11:49.279 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:11:49.279 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:11:49.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:49 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:11:47.632819+0000 mgr.smithi052.ghqrtk (mgr.14162) 63 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:51.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:51.018+0000 7f9f15c18700 1 -- 172.21.15.88:0/4127578162 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f100f6510 msgr2=0x7f9f100f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:51.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:51.018+0000 7f9f15c18700 1 --2- 172.21.15.88:0/4127578162 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f100f6510 0x7f9f100f6930 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f9f04009fb0 tx=0x7f9f04009b30).stop 2024-02-19T22:11:51.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:51.018+0000 7f9f15c18700 1 -- 172.21.15.88:0/4127578162 shutdown_connections 2024-02-19T22:11:51.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:51.018+0000 7f9f15c18700 1 --2- 172.21.15.88:0/4127578162 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f100f6510 0x7f9f100f6930 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:51.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:51.018+0000 7f9f15c18700 1 -- 172.21.15.88:0/4127578162 >> 172.21.15.88:0/4127578162 conn(0x7f9f100f1c70 msgr2=0x7f9f100f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:51.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:51.018+0000 7f9f15c18700 1 -- 172.21.15.88:0/4127578162 shutdown_connections 2024-02-19T22:11:51.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:51.018+0000 7f9f15c18700 1 -- 172.21.15.88:0/4127578162 wait complete. 2024-02-19T22:11:51.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:51.018+0000 7f9f15c18700 1 Processor -- start 2024-02-19T22:11:51.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:51.019+0000 7f9f15c18700 1 -- start start 2024-02-19T22:11:51.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:51.019+0000 7f9f15c18700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f100f6510 0x7f9f101067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:51.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:51.019+0000 7f9f15c18700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f10106cd0 con 0x7f9f100f6510 2024-02-19T22:11:51.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:51.019+0000 7f9f0f7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f100f6510 0x7f9f101067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:51.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:51.019+0000 7f9f0f7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f100f6510 0x7f9f101067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:33100/0 (socket says 172.21.15.88:33100) 2024-02-19T22:11:51.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:51.019+0000 7f9f0f7fe700 1 -- 172.21.15.88:0/2015092313 learned_addr learned my addr 172.21.15.88:0/2015092313 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:11:51.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:51.019+0000 7f9f0f7fe700 1 -- 172.21.15.88:0/2015092313 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f04005040 con 0x7f9f100f6510 2024-02-19T22:11:51.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:51.020+0000 7f9f0f7fe700 1 --2- 172.21.15.88:0/2015092313 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f100f6510 0x7f9f101067e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f9f04005010 tx=0x7f9f0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:51.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:51.020+0000 7f9ef7fff700 1 -- 172.21.15.88:0/2015092313 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f04009320 con 0x7f9f100f6510 2024-02-19T22:11:51.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:51.020+0000 7f9f15c18700 1 -- 172.21.15.88:0/2015092313 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f10106ed0 con 0x7f9f100f6510 2024-02-19T22:11:51.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:51.020+0000 7f9ef7fff700 1 -- 172.21.15.88:0/2015092313 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9f04013380 con 0x7f9f100f6510 2024-02-19T22:11:51.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:51.020+0000 7f9ef7fff700 1 -- 172.21.15.88:0/2015092313 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f04014e50 con 0x7f9f100f6510 2024-02-19T22:11:51.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:51.020+0000 7f9f15c18700 1 -- 172.21.15.88:0/2015092313 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f10107360 con 0x7f9f100f6510 2024-02-19T22:11:51.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:51.021+0000 7f9ef5ffb700 1 -- 172.21.15.88:0/2015092313 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ef0004fa0 con 0x7f9f100f6510 2024-02-19T22:11:51.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:51.021+0000 7f9ef7fff700 1 -- 172.21.15.88:0/2015092313 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9f04014420 con 0x7f9f100f6510 2024-02-19T22:11:51.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:51.021+0000 7f9ef7fff700 1 --2- 172.21.15.88:0/2015092313 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f9ef802ff10 0x7f9ef80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:51.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:51.021+0000 7f9ef7fff700 1 -- 172.21.15.88:0/2015092313 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9f04040f70 con 0x7f9f100f6510 2024-02-19T22:11:51.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:51.022+0000 7f9f0effd700 1 --2- 172.21.15.88:0/2015092313 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f9ef802ff10 0x7f9ef80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:51.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:51.022+0000 7f9f0effd700 1 --2- 172.21.15.88:0/2015092313 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f9ef802ff10 0x7f9ef80323d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f9f000096a0 tx=0x7f9f00006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:51.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:51.026+0000 7f9ef7fff700 1 -- 172.21.15.88:0/2015092313 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9f040095f0 con 0x7f9f100f6510 2024-02-19T22:11:51.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:51.230+0000 7f9ef5ffb700 1 -- 172.21.15.88:0/2015092313 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9ef0005670 con 0x7f9f100f6510 2024-02-19T22:11:51.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:51.230+0000 7f9ef7fff700 1 -- 172.21.15.88:0/2015092313 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9f0401db70 con 0x7f9f100f6510 2024-02-19T22:11:51.232 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:11:51.232 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:11:51.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:51.232+0000 7f9ef5ffb700 1 -- 172.21.15.88:0/2015092313 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f9ef802ff10 msgr2=0x7f9ef80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:51.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:51.232+0000 7f9ef5ffb700 1 --2- 172.21.15.88:0/2015092313 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f9ef802ff10 0x7f9ef80323d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f9f000096a0 tx=0x7f9f00006b40).stop 2024-02-19T22:11:51.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:51.232+0000 7f9ef5ffb700 1 -- 172.21.15.88:0/2015092313 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f100f6510 msgr2=0x7f9f101067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:51.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:51.232+0000 7f9ef5ffb700 1 --2- 172.21.15.88:0/2015092313 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f100f6510 0x7f9f101067e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f9f04005010 tx=0x7f9f0400b040).stop 2024-02-19T22:11:51.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:51.232+0000 7f9ef5ffb700 1 -- 172.21.15.88:0/2015092313 shutdown_connections 2024-02-19T22:11:51.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:51.232+0000 7f9ef5ffb700 1 --2- 172.21.15.88:0/2015092313 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f9ef802ff10 0x7f9ef80323d0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:51.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:51.232+0000 7f9ef5ffb700 1 --2- 172.21.15.88:0/2015092313 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f100f6510 0x7f9f101067e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:51.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:51.232+0000 7f9ef5ffb700 1 -- 172.21.15.88:0/2015092313 >> 172.21.15.88:0/2015092313 conn(0x7f9f100f1c70 msgr2=0x7f9f100f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:51.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:51.232+0000 7f9ef5ffb700 1 -- 172.21.15.88:0/2015092313 shutdown_connections 2024-02-19T22:11:51.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:51.232+0000 7f9ef5ffb700 1 -- 172.21.15.88:0/2015092313 wait complete. 2024-02-19T22:11:51.234 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:11:51.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:51 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:11:49.633300+0000 mgr.smithi052.ghqrtk (mgr.14162) 64 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:52.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:52 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:11:51.231862+0000 mon.smithi052 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.88:0/2015092313' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:11:52.872 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:11:52.872 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:11:53.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:53 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:11:51.633748+0000 mgr.smithi052.ghqrtk (mgr.14162) 65 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:54.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:54.613+0000 7fe17ce44700 1 -- 172.21.15.88:0/3067752782 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe1780f51a0 msgr2=0x7fe1780f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:54.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:54.613+0000 7fe17ce44700 1 --2- 172.21.15.88:0/3067752782 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe1780f51a0 0x7fe1780f55c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7fe168009fb0 tx=0x7fe168009b30).stop 2024-02-19T22:11:54.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:54.613+0000 7fe17ce44700 1 -- 172.21.15.88:0/3067752782 shutdown_connections 2024-02-19T22:11:54.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:54.613+0000 7fe17ce44700 1 --2- 172.21.15.88:0/3067752782 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe1780f51a0 0x7fe1780f55c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:54.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:54.613+0000 7fe17ce44700 1 -- 172.21.15.88:0/3067752782 >> 172.21.15.88:0/3067752782 conn(0x7fe1780f09a0 msgr2=0x7fe1780f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:54.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:54.614+0000 7fe17ce44700 1 -- 172.21.15.88:0/3067752782 shutdown_connections 2024-02-19T22:11:54.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:54.614+0000 7fe17ce44700 1 -- 172.21.15.88:0/3067752782 wait complete. 2024-02-19T22:11:54.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:54.614+0000 7fe17ce44700 1 Processor -- start 2024-02-19T22:11:54.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:54.614+0000 7fe17ce44700 1 -- start start 2024-02-19T22:11:54.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:54.615+0000 7fe17ce44700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe1780f51a0 0x7fe1781054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:54.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:54.615+0000 7fe17ce44700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1781059c0 con 0x7fe1780f51a0 2024-02-19T22:11:54.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:54.615+0000 7fe1777fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe1780f51a0 0x7fe1781054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:54.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:54.615+0000 7fe1777fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe1780f51a0 0x7fe1781054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:47446/0 (socket says 172.21.15.88:47446) 2024-02-19T22:11:54.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:54.615+0000 7fe1777fe700 1 -- 172.21.15.88:0/3606365981 learned_addr learned my addr 172.21.15.88:0/3606365981 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:11:54.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:54.615+0000 7fe1777fe700 1 -- 172.21.15.88:0/3606365981 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe168005040 con 0x7fe1780f51a0 2024-02-19T22:11:54.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:54.615+0000 7fe1777fe700 1 --2- 172.21.15.88:0/3606365981 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe1780f51a0 0x7fe1781054d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fe1680089f0 tx=0x7fe16800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:54.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:54.616+0000 7fe15ffff700 1 -- 172.21.15.88:0/3606365981 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe168009320 con 0x7fe1780f51a0 2024-02-19T22:11:54.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:54.616+0000 7fe17ce44700 1 -- 172.21.15.88:0/3606365981 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe178105bc0 con 0x7fe1780f51a0 2024-02-19T22:11:54.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:54.616+0000 7fe15ffff700 1 -- 172.21.15.88:0/3606365981 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe168013380 con 0x7fe1780f51a0 2024-02-19T22:11:54.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:54.616+0000 7fe15ffff700 1 -- 172.21.15.88:0/3606365981 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe168014e20 con 0x7fe1780f51a0 2024-02-19T22:11:54.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:54.616+0000 7fe17ce44700 1 -- 172.21.15.88:0/3606365981 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe178106050 con 0x7fe1780f51a0 2024-02-19T22:11:54.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:54.617+0000 7fe15ffff700 1 -- 172.21.15.88:0/3606365981 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe168014420 con 0x7fe1780f51a0 2024-02-19T22:11:54.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:54.617+0000 7fe15ffff700 1 --2- 172.21.15.88:0/3606365981 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fe16002ff10 0x7fe1600323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:54.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:54.617+0000 7fe15ffff700 1 -- 172.21.15.88:0/3606365981 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe168042d30 con 0x7fe1780f51a0 2024-02-19T22:11:54.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:54.617+0000 7fe17ce44700 1 -- 172.21.15.88:0/3606365981 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe178059100 con 0x7fe1780f51a0 2024-02-19T22:11:54.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:54.617+0000 7fe176ffd700 1 --2- 172.21.15.88:0/3606365981 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fe16002ff10 0x7fe1600323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:54.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:54.617+0000 7fe176ffd700 1 --2- 172.21.15.88:0/3606365981 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fe16002ff10 0x7fe1600323d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fe16c0096a0 tx=0x7fe16c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:54.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:54.622+0000 7fe15ffff700 1 -- 172.21.15.88:0/3606365981 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe1680095f0 con 0x7fe1780f51a0 2024-02-19T22:11:54.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:54.826+0000 7fe17ce44700 1 -- 172.21.15.88:0/3606365981 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe1780fad30 con 0x7fe1780f51a0 2024-02-19T22:11:54.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:54.827+0000 7fe15ffff700 1 -- 172.21.15.88:0/3606365981 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe16801fb70 con 0x7fe1780f51a0 2024-02-19T22:11:54.829 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:11:54.829 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:11:54.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:54.829+0000 7fe17ce44700 1 -- 172.21.15.88:0/3606365981 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fe16002ff10 msgr2=0x7fe1600323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:54.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:54.829+0000 7fe17ce44700 1 --2- 172.21.15.88:0/3606365981 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fe16002ff10 0x7fe1600323d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fe16c0096a0 tx=0x7fe16c006b40).stop 2024-02-19T22:11:54.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:54.829+0000 7fe17ce44700 1 -- 172.21.15.88:0/3606365981 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe1780f51a0 msgr2=0x7fe1781054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:54.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:54.829+0000 7fe17ce44700 1 --2- 172.21.15.88:0/3606365981 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe1780f51a0 0x7fe1781054d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fe1680089f0 tx=0x7fe16800b040).stop 2024-02-19T22:11:54.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:54.829+0000 7fe17ce44700 1 -- 172.21.15.88:0/3606365981 shutdown_connections 2024-02-19T22:11:54.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:54.829+0000 7fe17ce44700 1 --2- 172.21.15.88:0/3606365981 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fe16002ff10 0x7fe1600323d0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:54.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:54.829+0000 7fe17ce44700 1 --2- 172.21.15.88:0/3606365981 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe1780f51a0 0x7fe1781054d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:54.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:54.829+0000 7fe17ce44700 1 -- 172.21.15.88:0/3606365981 >> 172.21.15.88:0/3606365981 conn(0x7fe1780f09a0 msgr2=0x7fe1780f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:54.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:54.829+0000 7fe17ce44700 1 -- 172.21.15.88:0/3606365981 shutdown_connections 2024-02-19T22:11:54.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:54.829+0000 7fe17ce44700 1 -- 172.21.15.88:0/3606365981 wait complete. 2024-02-19T22:11:54.831 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:11:55.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:55 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:11:53.634162+0000 mgr.smithi052.ghqrtk (mgr.14162) 66 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:55.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:55 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:11:54.828364+0000 mon.smithi052 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.88:0/3606365981' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:11:56.490 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:11:56.491 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:11:57.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:57 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:11:55.634541+0000 mgr.smithi052.ghqrtk (mgr.14162) 67 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:58.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:58.348+0000 7f305ffff700 1 -- 172.21.15.88:0/2264563042 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30600f62a0 msgr2=0x7f30600f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:58.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:58.348+0000 7f305ffff700 1 --2- 172.21.15.88:0/2264563042 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30600f62a0 0x7f30600f66c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f3054009fb0 tx=0x7f3054009b30).stop 2024-02-19T22:11:58.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:58.348+0000 7f305ffff700 1 -- 172.21.15.88:0/2264563042 shutdown_connections 2024-02-19T22:11:58.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:58.348+0000 7f305ffff700 1 --2- 172.21.15.88:0/2264563042 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30600f62a0 0x7f30600f66c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:58.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:58.348+0000 7f305ffff700 1 -- 172.21.15.88:0/2264563042 >> 172.21.15.88:0/2264563042 conn(0x7f30600f1aa0 msgr2=0x7f30600f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:58.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:58.348+0000 7f305ffff700 1 -- 172.21.15.88:0/2264563042 shutdown_connections 2024-02-19T22:11:58.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:58.348+0000 7f305ffff700 1 -- 172.21.15.88:0/2264563042 wait complete. 2024-02-19T22:11:58.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:58.349+0000 7f305ffff700 1 Processor -- start 2024-02-19T22:11:58.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:58.349+0000 7f305ffff700 1 -- start start 2024-02-19T22:11:58.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:58.349+0000 7f305ffff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30600f62a0 0x7f30601065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:58.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:58.349+0000 7f305ffff700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3060106ac0 con 0x7f30600f62a0 2024-02-19T22:11:58.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:58.349+0000 7f305effd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30600f62a0 0x7f30601065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:58.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:58.349+0000 7f305effd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30600f62a0 0x7f30601065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:47462/0 (socket says 172.21.15.88:47462) 2024-02-19T22:11:58.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:58.349+0000 7f305effd700 1 -- 172.21.15.88:0/1534502901 learned_addr learned my addr 172.21.15.88:0/1534502901 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:11:58.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:58.350+0000 7f305effd700 1 -- 172.21.15.88:0/1534502901 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3054005040 con 0x7f30600f62a0 2024-02-19T22:11:58.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:58.350+0000 7f305effd700 1 --2- 172.21.15.88:0/1534502901 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30600f62a0 0x7f30601065d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f30540089f0 tx=0x7f305400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:58.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:58.350+0000 7f30477fe700 1 -- 172.21.15.88:0/1534502901 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3054009320 con 0x7f30600f62a0 2024-02-19T22:11:58.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:58.350+0000 7f30477fe700 1 -- 172.21.15.88:0/1534502901 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3054013380 con 0x7f30600f62a0 2024-02-19T22:11:58.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:58.350+0000 7f305ffff700 1 -- 172.21.15.88:0/1534502901 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3060106cc0 con 0x7f30600f62a0 2024-02-19T22:11:58.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:58.350+0000 7f30477fe700 1 -- 172.21.15.88:0/1534502901 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3054014e20 con 0x7f30600f62a0 2024-02-19T22:11:58.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:58.350+0000 7f305ffff700 1 -- 172.21.15.88:0/1534502901 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3060107150 con 0x7f30600f62a0 2024-02-19T22:11:58.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:58.351+0000 7f30477fe700 1 -- 172.21.15.88:0/1534502901 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f30540263a0 con 0x7f30600f62a0 2024-02-19T22:11:58.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:58.351+0000 7f30477fe700 1 --2- 172.21.15.88:0/1534502901 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f304802ff10 0x7f30480323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:58.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:58.351+0000 7f30477fe700 1 -- 172.21.15.88:0/1534502901 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3054040f10 con 0x7f30600f62a0 2024-02-19T22:11:58.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:58.351+0000 7f305ffff700 1 -- 172.21.15.88:0/1534502901 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3060059100 con 0x7f30600f62a0 2024-02-19T22:11:58.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:58.351+0000 7f305e7fc700 1 --2- 172.21.15.88:0/1534502901 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f304802ff10 0x7f30480323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:58.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:58.352+0000 7f305e7fc700 1 --2- 172.21.15.88:0/1534502901 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f304802ff10 0x7f30480323d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f30500096a0 tx=0x7f3050006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:58.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:58.356+0000 7f30477fe700 1 -- 172.21.15.88:0/1534502901 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3054012920 con 0x7f30600f62a0 2024-02-19T22:11:58.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:58.557+0000 7f305ffff700 1 -- 172.21.15.88:0/1534502901 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f30600fbe30 con 0x7f30600f62a0 2024-02-19T22:11:58.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:58.558+0000 7f30477fe700 1 -- 172.21.15.88:0/1534502901 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f305402b020 con 0x7f30600f62a0 2024-02-19T22:11:58.560 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:11:58.560 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:11:58.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:58.559+0000 7f305ffff700 1 -- 172.21.15.88:0/1534502901 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f304802ff10 msgr2=0x7f30480323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:58.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:58.559+0000 7f305ffff700 1 --2- 172.21.15.88:0/1534502901 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f304802ff10 0x7f30480323d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f30500096a0 tx=0x7f3050006b40).stop 2024-02-19T22:11:58.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:58.559+0000 7f305ffff700 1 -- 172.21.15.88:0/1534502901 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30600f62a0 msgr2=0x7f30601065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:58.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:58.559+0000 7f305ffff700 1 --2- 172.21.15.88:0/1534502901 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30600f62a0 0x7f30601065d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f30540089f0 tx=0x7f305400b040).stop 2024-02-19T22:11:58.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:58.559+0000 7f305ffff700 1 -- 172.21.15.88:0/1534502901 shutdown_connections 2024-02-19T22:11:58.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:58.559+0000 7f305ffff700 1 --2- 172.21.15.88:0/1534502901 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f304802ff10 0x7f30480323d0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:58.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:58.559+0000 7f305ffff700 1 --2- 172.21.15.88:0/1534502901 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30600f62a0 0x7f30601065d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:58.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:58.559+0000 7f305ffff700 1 -- 172.21.15.88:0/1534502901 >> 172.21.15.88:0/1534502901 conn(0x7f30600f1aa0 msgr2=0x7f30600f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:58.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:58.559+0000 7f305ffff700 1 -- 172.21.15.88:0/1534502901 shutdown_connections 2024-02-19T22:11:58.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:11:58.560+0000 7f305ffff700 1 -- 172.21.15.88:0/1534502901 wait complete. 2024-02-19T22:11:58.562 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:11:59.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:59 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:11:57.634749+0000 mgr.smithi052.ghqrtk (mgr.14162) 68 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:59.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:11:59 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:11:58.559390+0000 mon.smithi052 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.88:0/1534502901' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:12:00.162 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:12:00.162 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:12:01.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:01 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:11:59.635020+0000 mgr.smithi052.ghqrtk (mgr.14162) 69 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:01.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:01.946+0000 7f6efdc05700 1 -- 172.21.15.88:0/2035139197 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ef80f5410 msgr2=0x7f6ef80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:01.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:01.946+0000 7f6efdc05700 1 --2- 172.21.15.88:0/2035139197 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ef80f5410 0x7f6ef80f5830 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f6ee8009fb0 tx=0x7f6ee8009b30).stop 2024-02-19T22:12:01.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:01.946+0000 7f6efdc05700 1 -- 172.21.15.88:0/2035139197 shutdown_connections 2024-02-19T22:12:01.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:01.946+0000 7f6efdc05700 1 --2- 172.21.15.88:0/2035139197 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ef80f5410 0x7f6ef80f5830 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:01.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:01.946+0000 7f6efdc05700 1 -- 172.21.15.88:0/2035139197 >> 172.21.15.88:0/2035139197 conn(0x7f6ef80f0b70 msgr2=0x7f6ef80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:01.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:01.946+0000 7f6efdc05700 1 -- 172.21.15.88:0/2035139197 shutdown_connections 2024-02-19T22:12:01.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:01.946+0000 7f6efdc05700 1 -- 172.21.15.88:0/2035139197 wait complete. 2024-02-19T22:12:01.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:01.946+0000 7f6efdc05700 1 Processor -- start 2024-02-19T22:12:01.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:01.946+0000 7f6efdc05700 1 -- start start 2024-02-19T22:12:01.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:01.946+0000 7f6efdc05700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ef80f5410 0x7f6ef81094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:01.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:01.946+0000 7f6efdc05700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef8060240 con 0x7f6ef80f5410 2024-02-19T22:12:01.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:01.947+0000 7f6ef77fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ef80f5410 0x7f6ef81094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:01.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:01.947+0000 7f6ef77fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ef80f5410 0x7f6ef81094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:47466/0 (socket says 172.21.15.88:47466) 2024-02-19T22:12:01.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:01.947+0000 7f6ef77fe700 1 -- 172.21.15.88:0/177063663 learned_addr learned my addr 172.21.15.88:0/177063663 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:12:01.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:01.947+0000 7f6ef77fe700 1 -- 172.21.15.88:0/177063663 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee8005040 con 0x7f6ef80f5410 2024-02-19T22:12:01.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:01.947+0000 7f6ef77fe700 1 --2- 172.21.15.88:0/177063663 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ef80f5410 0x7f6ef81094e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f6ee8005010 tx=0x7f6ee800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:01.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:01.947+0000 7f6edffff700 1 -- 172.21.15.88:0/177063663 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ee8009320 con 0x7f6ef80f5410 2024-02-19T22:12:01.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:01.947+0000 7f6efdc05700 1 -- 172.21.15.88:0/177063663 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ef8060440 con 0x7f6ef80f5410 2024-02-19T22:12:01.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:01.947+0000 7f6edffff700 1 -- 172.21.15.88:0/177063663 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6ee8013380 con 0x7f6ef80f5410 2024-02-19T22:12:01.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:01.947+0000 7f6edffff700 1 -- 172.21.15.88:0/177063663 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ee8014e50 con 0x7f6ef80f5410 2024-02-19T22:12:01.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:01.947+0000 7f6efdc05700 1 -- 172.21.15.88:0/177063663 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ef8109b90 con 0x7f6ef80f5410 2024-02-19T22:12:01.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:01.948+0000 7f6edffff700 1 -- 172.21.15.88:0/177063663 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6ee80095f0 con 0x7f6ef80f5410 2024-02-19T22:12:01.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:01.948+0000 7f6efdc05700 1 -- 172.21.15.88:0/177063663 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ef8045ad0 con 0x7f6ef80f5410 2024-02-19T22:12:01.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:01.948+0000 7f6edffff700 1 --2- 172.21.15.88:0/177063663 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f6ee002ff10 0x7f6ee00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:01.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:01.948+0000 7f6edffff700 1 -- 172.21.15.88:0/177063663 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6ee8040fa0 con 0x7f6ef80f5410 2024-02-19T22:12:01.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:01.948+0000 7f6ef6ffd700 1 --2- 172.21.15.88:0/177063663 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f6ee002ff10 0x7f6ee00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:01.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:01.949+0000 7f6ef6ffd700 1 --2- 172.21.15.88:0/177063663 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f6ee002ff10 0x7f6ee00323d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f6eec0096a0 tx=0x7f6eec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:01.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:01.953+0000 7f6edffff700 1 -- 172.21.15.88:0/177063663 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6ee8014420 con 0x7f6ef80f5410 2024-02-19T22:12:02.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:02.157+0000 7f6efdc05700 1 -- 172.21.15.88:0/177063663 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6ef810a3b0 con 0x7f6ef80f5410 2024-02-19T22:12:02.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:02.158+0000 7f6edffff700 1 -- 172.21.15.88:0/177063663 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6ee8010230 con 0x7f6ef80f5410 2024-02-19T22:12:02.160 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:12:02.160 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:12:02.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:02.160+0000 7f6efdc05700 1 -- 172.21.15.88:0/177063663 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f6ee002ff10 msgr2=0x7f6ee00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:02.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:02.160+0000 7f6efdc05700 1 --2- 172.21.15.88:0/177063663 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f6ee002ff10 0x7f6ee00323d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f6eec0096a0 tx=0x7f6eec006b40).stop 2024-02-19T22:12:02.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:02.160+0000 7f6efdc05700 1 -- 172.21.15.88:0/177063663 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ef80f5410 msgr2=0x7f6ef81094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:02.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:02.160+0000 7f6efdc05700 1 --2- 172.21.15.88:0/177063663 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ef80f5410 0x7f6ef81094e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f6ee8005010 tx=0x7f6ee800b040).stop 2024-02-19T22:12:02.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:02.160+0000 7f6efdc05700 1 -- 172.21.15.88:0/177063663 shutdown_connections 2024-02-19T22:12:02.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:02.160+0000 7f6efdc05700 1 --2- 172.21.15.88:0/177063663 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f6ee002ff10 0x7f6ee00323d0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:02.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:02.160+0000 7f6efdc05700 1 --2- 172.21.15.88:0/177063663 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ef80f5410 0x7f6ef81094e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:02.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:02.160+0000 7f6efdc05700 1 -- 172.21.15.88:0/177063663 >> 172.21.15.88:0/177063663 conn(0x7f6ef80f0b70 msgr2=0x7f6ef80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:02.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:02.160+0000 7f6efdc05700 1 -- 172.21.15.88:0/177063663 shutdown_connections 2024-02-19T22:12:02.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:02.160+0000 7f6efdc05700 1 -- 172.21.15.88:0/177063663 wait complete. 2024-02-19T22:12:02.164 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:12:02.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:02 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:12:02.159509+0000 mon.smithi052 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.88:0/177063663' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:12:03.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:03 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:12:01.635344+0000 mgr.smithi052.ghqrtk (mgr.14162) 70 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:03.777 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:12:03.778 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:12:05.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:05.548+0000 7fd40bc5c700 1 -- 172.21.15.88:0/2811052605 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd4040f6480 msgr2=0x7fd4040f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:05.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:05.548+0000 7fd40bc5c700 1 --2- 172.21.15.88:0/2811052605 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd4040f6480 0x7fd4040f68a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fd3f4009fb0 tx=0x7fd3f4009b30).stop 2024-02-19T22:12:05.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:05.548+0000 7fd40bc5c700 1 -- 172.21.15.88:0/2811052605 shutdown_connections 2024-02-19T22:12:05.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:05.548+0000 7fd40bc5c700 1 --2- 172.21.15.88:0/2811052605 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd4040f6480 0x7fd4040f68a0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:05.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:05.548+0000 7fd40bc5c700 1 -- 172.21.15.88:0/2811052605 >> 172.21.15.88:0/2811052605 conn(0x7fd4040f1c00 msgr2=0x7fd4040f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:05.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:05.548+0000 7fd40bc5c700 1 -- 172.21.15.88:0/2811052605 shutdown_connections 2024-02-19T22:12:05.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:05.548+0000 7fd40bc5c700 1 -- 172.21.15.88:0/2811052605 wait complete. 2024-02-19T22:12:05.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:05.548+0000 7fd40bc5c700 1 Processor -- start 2024-02-19T22:12:05.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:05.548+0000 7fd40bc5c700 1 -- start start 2024-02-19T22:12:05.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:05.548+0000 7fd40bc5c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd4040f6480 0x7fd40406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:05.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:05.548+0000 7fd40bc5c700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd404069e20 con 0x7fd4040f6480 2024-02-19T22:12:05.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:05.549+0000 7fd4099f8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd4040f6480 0x7fd40406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:05.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:05.549+0000 7fd4099f8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd4040f6480 0x7fd40406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:43802/0 (socket says 172.21.15.88:43802) 2024-02-19T22:12:05.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:05.549+0000 7fd4099f8700 1 -- 172.21.15.88:0/3714583991 learned_addr learned my addr 172.21.15.88:0/3714583991 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:12:05.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:05.549+0000 7fd4099f8700 1 -- 172.21.15.88:0/3714583991 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f4005040 con 0x7fd4040f6480 2024-02-19T22:12:05.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:05.549+0000 7fd4099f8700 1 --2- 172.21.15.88:0/3714583991 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd4040f6480 0x7fd40406b9a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fd3f4005010 tx=0x7fd3f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:05.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:05.549+0000 7fd3fa7fc700 1 -- 172.21.15.88:0/3714583991 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3f4009320 con 0x7fd4040f6480 2024-02-19T22:12:05.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:05.549+0000 7fd40bc5c700 1 -- 172.21.15.88:0/3714583991 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd404069fc0 con 0x7fd4040f6480 2024-02-19T22:12:05.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:05.549+0000 7fd3fa7fc700 1 -- 172.21.15.88:0/3714583991 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd3f4013380 con 0x7fd4040f6480 2024-02-19T22:12:05.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:05.549+0000 7fd3fa7fc700 1 -- 172.21.15.88:0/3714583991 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3f4014e50 con 0x7fd4040f6480 2024-02-19T22:12:05.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:05.549+0000 7fd40bc5c700 1 -- 172.21.15.88:0/3714583991 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd40406a450 con 0x7fd4040f6480 2024-02-19T22:12:05.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:05.550+0000 7fd3fa7fc700 1 -- 172.21.15.88:0/3714583991 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd3f4014420 con 0x7fd4040f6480 2024-02-19T22:12:05.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:05.550+0000 7fd40bc5c700 1 -- 172.21.15.88:0/3714583991 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd404059100 con 0x7fd4040f6480 2024-02-19T22:12:05.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:05.550+0000 7fd3fa7fc700 1 --2- 172.21.15.88:0/3714583991 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd3f002ff10 0x7fd3f00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:05.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:05.550+0000 7fd3fa7fc700 1 -- 172.21.15.88:0/3714583991 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd3f4040f40 con 0x7fd4040f6480 2024-02-19T22:12:05.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:05.550+0000 7fd4091f7700 1 --2- 172.21.15.88:0/3714583991 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd3f002ff10 0x7fd3f00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:05.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:05.551+0000 7fd4091f7700 1 --2- 172.21.15.88:0/3714583991 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd3f002ff10 0x7fd3f00323d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fd4000096a0 tx=0x7fd400006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:05.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:05.555+0000 7fd3fa7fc700 1 -- 172.21.15.88:0/3714583991 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd3f40095f0 con 0x7fd4040f6480 2024-02-19T22:12:05.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:05 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:12:03.635657+0000 mgr.smithi052.ghqrtk (mgr.14162) 71 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:05.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:05.759+0000 7fd40bc5c700 1 -- 172.21.15.88:0/3714583991 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd4040fc090 con 0x7fd4040f6480 2024-02-19T22:12:05.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:05.759+0000 7fd3fa7fc700 1 -- 172.21.15.88:0/3714583991 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd3f401db70 con 0x7fd4040f6480 2024-02-19T22:12:05.761 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:12:05.761 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:12:05.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:05.761+0000 7fd40bc5c700 1 -- 172.21.15.88:0/3714583991 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd3f002ff10 msgr2=0x7fd3f00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:05.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:05.761+0000 7fd40bc5c700 1 --2- 172.21.15.88:0/3714583991 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd3f002ff10 0x7fd3f00323d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fd4000096a0 tx=0x7fd400006b40).stop 2024-02-19T22:12:05.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:05.761+0000 7fd40bc5c700 1 -- 172.21.15.88:0/3714583991 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd4040f6480 msgr2=0x7fd40406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:05.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:05.761+0000 7fd40bc5c700 1 --2- 172.21.15.88:0/3714583991 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd4040f6480 0x7fd40406b9a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fd3f4005010 tx=0x7fd3f400b040).stop 2024-02-19T22:12:05.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:05.761+0000 7fd40bc5c700 1 -- 172.21.15.88:0/3714583991 shutdown_connections 2024-02-19T22:12:05.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:05.761+0000 7fd40bc5c700 1 --2- 172.21.15.88:0/3714583991 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd3f002ff10 0x7fd3f00323d0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:05.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:05.761+0000 7fd40bc5c700 1 --2- 172.21.15.88:0/3714583991 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd4040f6480 0x7fd40406b9a0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:05.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:05.761+0000 7fd40bc5c700 1 -- 172.21.15.88:0/3714583991 >> 172.21.15.88:0/3714583991 conn(0x7fd4040f1c00 msgr2=0x7fd4040f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:05.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:05.761+0000 7fd40bc5c700 1 -- 172.21.15.88:0/3714583991 shutdown_connections 2024-02-19T22:12:05.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:05.761+0000 7fd40bc5c700 1 -- 172.21.15.88:0/3714583991 wait complete. 2024-02-19T22:12:05.763 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:12:06.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:06 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:12:05.760826+0000 mon.smithi052 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.88:0/3714583991' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:12:07.562 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:12:07.562 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:12:07.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:07 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:12:05.635953+0000 mgr.smithi052.ghqrtk (mgr.14162) 72 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:09.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:09.194+0000 7f6c46020700 1 -- 172.21.15.88:0/2245096122 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6c400f5410 msgr2=0x7f6c400f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:09.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:09.194+0000 7f6c46020700 1 --2- 172.21.15.88:0/2245096122 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6c400f5410 0x7f6c400f5830 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f6c30009fb0 tx=0x7f6c30009b30).stop 2024-02-19T22:12:09.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:09.194+0000 7f6c46020700 1 -- 172.21.15.88:0/2245096122 shutdown_connections 2024-02-19T22:12:09.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:09.194+0000 7f6c46020700 1 --2- 172.21.15.88:0/2245096122 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6c400f5410 0x7f6c400f5830 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:09.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:09.194+0000 7f6c46020700 1 -- 172.21.15.88:0/2245096122 >> 172.21.15.88:0/2245096122 conn(0x7f6c400f0b70 msgr2=0x7f6c400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:09.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:09.194+0000 7f6c46020700 1 -- 172.21.15.88:0/2245096122 shutdown_connections 2024-02-19T22:12:09.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:09.194+0000 7f6c46020700 1 -- 172.21.15.88:0/2245096122 wait complete. 2024-02-19T22:12:09.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:09.194+0000 7f6c46020700 1 Processor -- start 2024-02-19T22:12:09.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:09.194+0000 7f6c46020700 1 -- start start 2024-02-19T22:12:09.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:09.194+0000 7f6c46020700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6c400f5410 0x7f6c401056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:09.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:09.194+0000 7f6c46020700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c40105bd0 con 0x7f6c400f5410 2024-02-19T22:12:09.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:09.195+0000 7f6c3f7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6c400f5410 0x7f6c401056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:09.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:09.195+0000 7f6c3f7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6c400f5410 0x7f6c401056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:43826/0 (socket says 172.21.15.88:43826) 2024-02-19T22:12:09.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:09.195+0000 7f6c3f7fe700 1 -- 172.21.15.88:0/3791374854 learned_addr learned my addr 172.21.15.88:0/3791374854 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:12:09.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:09.195+0000 7f6c3f7fe700 1 -- 172.21.15.88:0/3791374854 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c30005040 con 0x7f6c400f5410 2024-02-19T22:12:09.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:09.195+0000 7f6c3f7fe700 1 --2- 172.21.15.88:0/3791374854 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6c400f5410 0x7f6c401056e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f6c300089f0 tx=0x7f6c3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:09.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:09.195+0000 7f6c27fff700 1 -- 172.21.15.88:0/3791374854 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c30009320 con 0x7f6c400f5410 2024-02-19T22:12:09.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:09.195+0000 7f6c46020700 1 -- 172.21.15.88:0/3791374854 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c40105dd0 con 0x7f6c400f5410 2024-02-19T22:12:09.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:09.195+0000 7f6c27fff700 1 -- 172.21.15.88:0/3791374854 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6c30013380 con 0x7f6c400f5410 2024-02-19T22:12:09.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:09.195+0000 7f6c27fff700 1 -- 172.21.15.88:0/3791374854 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c3001ed80 con 0x7f6c400f5410 2024-02-19T22:12:09.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:09.195+0000 7f6c46020700 1 -- 172.21.15.88:0/3791374854 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c40106260 con 0x7f6c400f5410 2024-02-19T22:12:09.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:09.196+0000 7f6c27fff700 1 -- 172.21.15.88:0/3791374854 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6c3001e420 con 0x7f6c400f5410 2024-02-19T22:12:09.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:09.196+0000 7f6c46020700 1 -- 172.21.15.88:0/3791374854 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c40045ad0 con 0x7f6c400f5410 2024-02-19T22:12:09.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:09.196+0000 7f6c27fff700 1 --2- 172.21.15.88:0/3791374854 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f6c2802ff60 0x7f6c28032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:09.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:09.196+0000 7f6c27fff700 1 -- 172.21.15.88:0/3791374854 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6c300410f0 con 0x7f6c400f5410 2024-02-19T22:12:09.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:09.197+0000 7f6c3effd700 1 --2- 172.21.15.88:0/3791374854 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f6c2802ff60 0x7f6c28032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:09.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:09.197+0000 7f6c3effd700 1 --2- 172.21.15.88:0/3791374854 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f6c2802ff60 0x7f6c28032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f6c340096a0 tx=0x7f6c34006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:09.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:09.201+0000 7f6c27fff700 1 -- 172.21.15.88:0/3791374854 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6c3001cb10 con 0x7f6c400f5410 2024-02-19T22:12:09.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:09.406+0000 7f6c46020700 1 -- 172.21.15.88:0/3791374854 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6c401070b0 con 0x7f6c400f5410 2024-02-19T22:12:09.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:09.406+0000 7f6c27fff700 1 -- 172.21.15.88:0/3791374854 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6c3002c2d0 con 0x7f6c400f5410 2024-02-19T22:12:09.408 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:12:09.408 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:12:09.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:09.408+0000 7f6c46020700 1 -- 172.21.15.88:0/3791374854 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f6c2802ff60 msgr2=0x7f6c28032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:09.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:09.408+0000 7f6c46020700 1 --2- 172.21.15.88:0/3791374854 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f6c2802ff60 0x7f6c28032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f6c340096a0 tx=0x7f6c34006b40).stop 2024-02-19T22:12:09.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:09.408+0000 7f6c46020700 1 -- 172.21.15.88:0/3791374854 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6c400f5410 msgr2=0x7f6c401056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:09.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:09.408+0000 7f6c46020700 1 --2- 172.21.15.88:0/3791374854 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6c400f5410 0x7f6c401056e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f6c300089f0 tx=0x7f6c3000b040).stop 2024-02-19T22:12:09.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:09.408+0000 7f6c46020700 1 -- 172.21.15.88:0/3791374854 shutdown_connections 2024-02-19T22:12:09.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:09.408+0000 7f6c46020700 1 --2- 172.21.15.88:0/3791374854 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f6c2802ff60 0x7f6c28032420 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:09.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:09.408+0000 7f6c46020700 1 --2- 172.21.15.88:0/3791374854 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6c400f5410 0x7f6c401056e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:09.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:09.408+0000 7f6c46020700 1 -- 172.21.15.88:0/3791374854 >> 172.21.15.88:0/3791374854 conn(0x7f6c400f0b70 msgr2=0x7f6c400f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:09.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:09.408+0000 7f6c46020700 1 -- 172.21.15.88:0/3791374854 shutdown_connections 2024-02-19T22:12:09.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:09.408+0000 7f6c46020700 1 -- 172.21.15.88:0/3791374854 wait complete. 2024-02-19T22:12:09.411 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:12:09.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:09 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:12:07.636258+0000 mgr.smithi052.ghqrtk (mgr.14162) 73 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:10.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:10 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:12:09.407756+0000 mon.smithi052 (mon.0) 150 : audit [DBG] from='client.? 172.21.15.88:0/3791374854' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:12:11.001 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:12:11.001 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:12:11.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:11 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:12:09.636553+0000 mgr.smithi052.ghqrtk (mgr.14162) 74 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:12.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:12.807+0000 7f36ee765700 1 -- 172.21.15.88:0/3531399907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f36e806b960 msgr2=0x7f36e8069e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:12.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:12.807+0000 7f36ee765700 1 --2- 172.21.15.88:0/3531399907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f36e806b960 0x7f36e8069e60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f36d8009fb0 tx=0x7f36d8009b30).stop 2024-02-19T22:12:12.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:12.807+0000 7f36ee765700 1 -- 172.21.15.88:0/3531399907 shutdown_connections 2024-02-19T22:12:12.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:12.807+0000 7f36ee765700 1 --2- 172.21.15.88:0/3531399907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f36e806b960 0x7f36e8069e60 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:12.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:12.807+0000 7f36ee765700 1 -- 172.21.15.88:0/3531399907 >> 172.21.15.88:0/3531399907 conn(0x7f36e80f1c70 msgr2=0x7f36e80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:12.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:12.807+0000 7f36ee765700 1 -- 172.21.15.88:0/3531399907 shutdown_connections 2024-02-19T22:12:12.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:12.807+0000 7f36ee765700 1 -- 172.21.15.88:0/3531399907 wait complete. 2024-02-19T22:12:12.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:12.808+0000 7f36ee765700 1 Processor -- start 2024-02-19T22:12:12.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:12.808+0000 7f36ee765700 1 -- start start 2024-02-19T22:12:12.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:12.808+0000 7f36ee765700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f36e806b960 0x7f36e81089a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:12.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:12.808+0000 7f36ee765700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e8108e90 con 0x7f36e806b960 2024-02-19T22:12:12.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:12.808+0000 7f36e7fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f36e806b960 0x7f36e81089a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:12.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:12.808+0000 7f36e7fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f36e806b960 0x7f36e81089a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:43846/0 (socket says 172.21.15.88:43846) 2024-02-19T22:12:12.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:12.808+0000 7f36e7fff700 1 -- 172.21.15.88:0/2660749686 learned_addr learned my addr 172.21.15.88:0/2660749686 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:12:12.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:12.808+0000 7f36e7fff700 1 -- 172.21.15.88:0/2660749686 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d8005040 con 0x7f36e806b960 2024-02-19T22:12:12.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:12.809+0000 7f36e7fff700 1 --2- 172.21.15.88:0/2660749686 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f36e806b960 0x7f36e81089a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f36d80089f0 tx=0x7f36d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:12.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:12.809+0000 7f36e4ff9700 1 -- 172.21.15.88:0/2660749686 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f36d8009320 con 0x7f36e806b960 2024-02-19T22:12:12.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:12.809+0000 7f36ee765700 1 -- 172.21.15.88:0/2660749686 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36e8109090 con 0x7f36e806b960 2024-02-19T22:12:12.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:12.809+0000 7f36e4ff9700 1 -- 172.21.15.88:0/2660749686 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f36d8013380 con 0x7f36e806b960 2024-02-19T22:12:12.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:12.809+0000 7f36ee765700 1 -- 172.21.15.88:0/2660749686 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36e8109520 con 0x7f36e806b960 2024-02-19T22:12:12.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:12.809+0000 7f36e4ff9700 1 -- 172.21.15.88:0/2660749686 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f36d8014e20 con 0x7f36e806b960 2024-02-19T22:12:12.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:12.810+0000 7f36e4ff9700 1 -- 172.21.15.88:0/2660749686 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f36d8014420 con 0x7f36e806b960 2024-02-19T22:12:12.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:12.810+0000 7f36ee765700 1 -- 172.21.15.88:0/2660749686 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36e8045ad0 con 0x7f36e806b960 2024-02-19T22:12:12.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:12.810+0000 7f36e4ff9700 1 --2- 172.21.15.88:0/2660749686 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f36d002ff10 0x7f36d00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:12.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:12.810+0000 7f36e4ff9700 1 -- 172.21.15.88:0/2660749686 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f36d8040eb0 con 0x7f36e806b960 2024-02-19T22:12:12.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:12.810+0000 7f36e77fe700 1 --2- 172.21.15.88:0/2660749686 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f36d002ff10 0x7f36d00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:12.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:12.811+0000 7f36e77fe700 1 --2- 172.21.15.88:0/2660749686 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f36d002ff10 0x7f36d00323d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f36dc0096a0 tx=0x7f36dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:12.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:12.814+0000 7f36e4ff9700 1 -- 172.21.15.88:0/2660749686 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f36d8014790 con 0x7f36e806b960 2024-02-19T22:12:13.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:13.017+0000 7f36ee765700 1 -- 172.21.15.88:0/2660749686 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f36e810a370 con 0x7f36e806b960 2024-02-19T22:12:13.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:13.017+0000 7f36e4ff9700 1 -- 172.21.15.88:0/2660749686 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f36d8010230 con 0x7f36e806b960 2024-02-19T22:12:13.019 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:12:13.019 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:12:13.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:13.019+0000 7f36ee765700 1 -- 172.21.15.88:0/2660749686 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f36d002ff10 msgr2=0x7f36d00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:13.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:13.019+0000 7f36ee765700 1 --2- 172.21.15.88:0/2660749686 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f36d002ff10 0x7f36d00323d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f36dc0096a0 tx=0x7f36dc006b40).stop 2024-02-19T22:12:13.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:13.019+0000 7f36ee765700 1 -- 172.21.15.88:0/2660749686 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f36e806b960 msgr2=0x7f36e81089a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:13.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:13.019+0000 7f36ee765700 1 --2- 172.21.15.88:0/2660749686 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f36e806b960 0x7f36e81089a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f36d80089f0 tx=0x7f36d800b040).stop 2024-02-19T22:12:13.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:13.019+0000 7f36ee765700 1 -- 172.21.15.88:0/2660749686 shutdown_connections 2024-02-19T22:12:13.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:13.019+0000 7f36ee765700 1 --2- 172.21.15.88:0/2660749686 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f36d002ff10 0x7f36d00323d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:13.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:13.019+0000 7f36ee765700 1 --2- 172.21.15.88:0/2660749686 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f36e806b960 0x7f36e81089a0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:13.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:13.019+0000 7f36ee765700 1 -- 172.21.15.88:0/2660749686 >> 172.21.15.88:0/2660749686 conn(0x7f36e80f1c70 msgr2=0x7f36e80f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:13.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:13.019+0000 7f36ee765700 1 -- 172.21.15.88:0/2660749686 shutdown_connections 2024-02-19T22:12:13.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:13.019+0000 7f36ee765700 1 -- 172.21.15.88:0/2660749686 wait complete. 2024-02-19T22:12:13.022 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:12:13.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:13 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:12:11.636950+0000 mgr.smithi052.ghqrtk (mgr.14162) 75 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:13.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:13 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:12:13.019018+0000 mon.smithi052 (mon.0) 151 : audit [DBG] from='client.? 172.21.15.88:0/2660749686' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:12:14.633 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:12:14.634 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:12:15.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:15 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:12:13.637434+0000 mgr.smithi052.ghqrtk (mgr.14162) 76 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:16.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:16.452+0000 7f18ddd6f700 1 -- 172.21.15.88:0/2446012561 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f18d80f62a0 msgr2=0x7f18d80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:16.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:16.452+0000 7f18ddd6f700 1 --2- 172.21.15.88:0/2446012561 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f18d80f62a0 0x7f18d80f66c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f18cc009fb0 tx=0x7f18cc009b30).stop 2024-02-19T22:12:16.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:16.452+0000 7f18ddd6f700 1 -- 172.21.15.88:0/2446012561 shutdown_connections 2024-02-19T22:12:16.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:16.452+0000 7f18ddd6f700 1 --2- 172.21.15.88:0/2446012561 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f18d80f62a0 0x7f18d80f66c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:16.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:16.452+0000 7f18ddd6f700 1 -- 172.21.15.88:0/2446012561 >> 172.21.15.88:0/2446012561 conn(0x7f18d80f1aa0 msgr2=0x7f18d80f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:16.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:16.452+0000 7f18ddd6f700 1 -- 172.21.15.88:0/2446012561 shutdown_connections 2024-02-19T22:12:16.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:16.452+0000 7f18ddd6f700 1 -- 172.21.15.88:0/2446012561 wait complete. 2024-02-19T22:12:16.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:16.452+0000 7f18ddd6f700 1 Processor -- start 2024-02-19T22:12:16.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:16.452+0000 7f18ddd6f700 1 -- start start 2024-02-19T22:12:16.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:16.453+0000 7f18ddd6f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f18d80f62a0 0x7f18d81065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:16.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:16.453+0000 7f18ddd6f700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d8106ac0 con 0x7f18d80f62a0 2024-02-19T22:12:16.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:16.453+0000 7f18dcd6d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f18d80f62a0 0x7f18d81065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:16.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:16.453+0000 7f18dcd6d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f18d80f62a0 0x7f18d81065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:38682/0 (socket says 172.21.15.88:38682) 2024-02-19T22:12:16.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:16.453+0000 7f18dcd6d700 1 -- 172.21.15.88:0/2057860808 learned_addr learned my addr 172.21.15.88:0/2057860808 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:12:16.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:16.453+0000 7f18dcd6d700 1 -- 172.21.15.88:0/2057860808 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18cc005040 con 0x7f18d80f62a0 2024-02-19T22:12:16.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:16.453+0000 7f18dcd6d700 1 --2- 172.21.15.88:0/2057860808 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f18d80f62a0 0x7f18d81065d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f18cc0089f0 tx=0x7f18cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:16.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:16.454+0000 7f18d57fa700 1 -- 172.21.15.88:0/2057860808 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18cc009320 con 0x7f18d80f62a0 2024-02-19T22:12:16.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:16.454+0000 7f18ddd6f700 1 -- 172.21.15.88:0/2057860808 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18d8106cc0 con 0x7f18d80f62a0 2024-02-19T22:12:16.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:16.454+0000 7f18d57fa700 1 -- 172.21.15.88:0/2057860808 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f18cc013380 con 0x7f18d80f62a0 2024-02-19T22:12:16.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:16.454+0000 7f18d57fa700 1 -- 172.21.15.88:0/2057860808 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18cc014e20 con 0x7f18d80f62a0 2024-02-19T22:12:16.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:16.454+0000 7f18ddd6f700 1 -- 172.21.15.88:0/2057860808 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18d8107150 con 0x7f18d80f62a0 2024-02-19T22:12:16.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:16.454+0000 7f18d57fa700 1 -- 172.21.15.88:0/2057860808 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f18cc014420 con 0x7f18d80f62a0 2024-02-19T22:12:16.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:16.454+0000 7f18ddd6f700 1 -- 172.21.15.88:0/2057860808 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18d8045ad0 con 0x7f18d80f62a0 2024-02-19T22:12:16.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:16.455+0000 7f18d57fa700 1 --2- 172.21.15.88:0/2057860808 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f18c802ff10 0x7f18c80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:16.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:16.455+0000 7f18d57fa700 1 -- 172.21.15.88:0/2057860808 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f18cc040fc0 con 0x7f18d80f62a0 2024-02-19T22:12:16.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:16.455+0000 7f18d7fff700 1 --2- 172.21.15.88:0/2057860808 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f18c802ff10 0x7f18c80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:16.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:16.455+0000 7f18d7fff700 1 --2- 172.21.15.88:0/2057860808 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f18c802ff10 0x7f18c80323d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f18c40096a0 tx=0x7f18c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:16.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:16.460+0000 7f18d57fa700 1 -- 172.21.15.88:0/2057860808 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f18cc06d2b0 con 0x7f18d80f62a0 2024-02-19T22:12:16.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:16.662+0000 7f18ddd6f700 1 -- 172.21.15.88:0/2057860808 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f18d8108000 con 0x7f18d80f62a0 2024-02-19T22:12:16.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:16.663+0000 7f18d57fa700 1 -- 172.21.15.88:0/2057860808 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f18cc0095f0 con 0x7f18d80f62a0 2024-02-19T22:12:16.665 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:12:16.665 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:12:16.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:16.665+0000 7f18ddd6f700 1 -- 172.21.15.88:0/2057860808 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f18c802ff10 msgr2=0x7f18c80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:16.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:16.665+0000 7f18ddd6f700 1 --2- 172.21.15.88:0/2057860808 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f18c802ff10 0x7f18c80323d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f18c40096a0 tx=0x7f18c4006b40).stop 2024-02-19T22:12:16.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:16.665+0000 7f18ddd6f700 1 -- 172.21.15.88:0/2057860808 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f18d80f62a0 msgr2=0x7f18d81065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:16.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:16.665+0000 7f18ddd6f700 1 --2- 172.21.15.88:0/2057860808 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f18d80f62a0 0x7f18d81065d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f18cc0089f0 tx=0x7f18cc00b040).stop 2024-02-19T22:12:16.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:16.665+0000 7f18ddd6f700 1 -- 172.21.15.88:0/2057860808 shutdown_connections 2024-02-19T22:12:16.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:16.665+0000 7f18ddd6f700 1 --2- 172.21.15.88:0/2057860808 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f18c802ff10 0x7f18c80323d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:16.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:16.665+0000 7f18ddd6f700 1 --2- 172.21.15.88:0/2057860808 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f18d80f62a0 0x7f18d81065d0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:16.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:16.665+0000 7f18ddd6f700 1 -- 172.21.15.88:0/2057860808 >> 172.21.15.88:0/2057860808 conn(0x7f18d80f1aa0 msgr2=0x7f18d80f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:16.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:16.665+0000 7f18ddd6f700 1 -- 172.21.15.88:0/2057860808 shutdown_connections 2024-02-19T22:12:16.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:16.665+0000 7f18ddd6f700 1 -- 172.21.15.88:0/2057860808 wait complete. 2024-02-19T22:12:16.667 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:12:17.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:17 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:12:15.637753+0000 mgr.smithi052.ghqrtk (mgr.14162) 77 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:17.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:17 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:12:16.664524+0000 mon.smithi052 (mon.0) 152 : audit [DBG] from='client.? 172.21.15.88:0/2057860808' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:12:18.272 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:12:18.272 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:12:19.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:19 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:12:17.638015+0000 mgr.smithi052.ghqrtk (mgr.14162) 78 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:19.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:19.996+0000 7ff7ab439700 1 -- 172.21.15.88:0/3112828347 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff7a40f5410 msgr2=0x7ff7a40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:19.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:19.996+0000 7ff7ab439700 1 --2- 172.21.15.88:0/3112828347 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff7a40f5410 0x7ff7a40f5830 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7ff798009fb0 tx=0x7ff798009b30).stop 2024-02-19T22:12:19.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:19.996+0000 7ff7ab439700 1 -- 172.21.15.88:0/3112828347 shutdown_connections 2024-02-19T22:12:19.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:19.996+0000 7ff7ab439700 1 --2- 172.21.15.88:0/3112828347 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff7a40f5410 0x7ff7a40f5830 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:19.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:19.996+0000 7ff7ab439700 1 -- 172.21.15.88:0/3112828347 >> 172.21.15.88:0/3112828347 conn(0x7ff7a40f0b70 msgr2=0x7ff7a40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:19.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:19.996+0000 7ff7ab439700 1 -- 172.21.15.88:0/3112828347 shutdown_connections 2024-02-19T22:12:19.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:19.996+0000 7ff7ab439700 1 -- 172.21.15.88:0/3112828347 wait complete. 2024-02-19T22:12:19.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:19.997+0000 7ff7ab439700 1 Processor -- start 2024-02-19T22:12:19.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:19.997+0000 7ff7ab439700 1 -- start start 2024-02-19T22:12:19.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:19.997+0000 7ff7ab439700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff7a40f5410 0x7ff7a41067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:19.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:19.997+0000 7ff7ab439700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a4106cd0 con 0x7ff7a40f5410 2024-02-19T22:12:19.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:19.997+0000 7ff7a91d5700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff7a40f5410 0x7ff7a41067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:19.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:19.998+0000 7ff7a91d5700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff7a40f5410 0x7ff7a41067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:38704/0 (socket says 172.21.15.88:38704) 2024-02-19T22:12:19.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:19.998+0000 7ff7a91d5700 1 -- 172.21.15.88:0/108118091 learned_addr learned my addr 172.21.15.88:0/108118091 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:12:19.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:19.998+0000 7ff7a91d5700 1 -- 172.21.15.88:0/108118091 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff798005040 con 0x7ff7a40f5410 2024-02-19T22:12:19.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:19.998+0000 7ff7a91d5700 1 --2- 172.21.15.88:0/108118091 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff7a40f5410 0x7ff7a41067e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7ff798005010 tx=0x7ff79800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:20.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:19.998+0000 7ff795ffb700 1 -- 172.21.15.88:0/108118091 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff798009320 con 0x7ff7a40f5410 2024-02-19T22:12:20.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:19.998+0000 7ff7ab439700 1 -- 172.21.15.88:0/108118091 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7a4106ed0 con 0x7ff7a40f5410 2024-02-19T22:12:20.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:19.998+0000 7ff795ffb700 1 -- 172.21.15.88:0/108118091 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff798013380 con 0x7ff7a40f5410 2024-02-19T22:12:20.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:19.998+0000 7ff795ffb700 1 -- 172.21.15.88:0/108118091 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff798014e50 con 0x7ff7a40f5410 2024-02-19T22:12:20.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:19.998+0000 7ff7ab439700 1 -- 172.21.15.88:0/108118091 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7a4107360 con 0x7ff7a40f5410 2024-02-19T22:12:20.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:19.999+0000 7ff795ffb700 1 -- 172.21.15.88:0/108118091 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff7980263a0 con 0x7ff7a40f5410 2024-02-19T22:12:20.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:19.999+0000 7ff795ffb700 1 --2- 172.21.15.88:0/108118091 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7ff79002ff10 0x7ff7900323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:20.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:19.999+0000 7ff795ffb700 1 -- 172.21.15.88:0/108118091 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff7980095f0 con 0x7ff7a40f5410 2024-02-19T22:12:20.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:19.999+0000 7ff7ab439700 1 -- 172.21.15.88:0/108118091 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7a4059100 con 0x7ff7a40f5410 2024-02-19T22:12:20.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:20.000+0000 7ff7a89d4700 1 --2- 172.21.15.88:0/108118091 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7ff79002ff10 0x7ff7900323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:20.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:20.000+0000 7ff7a89d4700 1 --2- 172.21.15.88:0/108118091 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7ff79002ff10 0x7ff7900323d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7ff7a00096a0 tx=0x7ff7a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:20.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:20.004+0000 7ff795ffb700 1 -- 172.21.15.88:0/108118091 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff798012920 con 0x7ff7a40f5410 2024-02-19T22:12:20.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:20.208+0000 7ff7ab439700 1 -- 172.21.15.88:0/108118091 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff7a40fafa0 con 0x7ff7a40f5410 2024-02-19T22:12:20.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:20.209+0000 7ff795ffb700 1 -- 172.21.15.88:0/108118091 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff79802b030 con 0x7ff7a40f5410 2024-02-19T22:12:20.209 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:12:20.209 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:12:20.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:20.210+0000 7ff7ab439700 1 -- 172.21.15.88:0/108118091 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7ff79002ff10 msgr2=0x7ff7900323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:20.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:20.210+0000 7ff7ab439700 1 --2- 172.21.15.88:0/108118091 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7ff79002ff10 0x7ff7900323d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7ff7a00096a0 tx=0x7ff7a0006b40).stop 2024-02-19T22:12:20.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:20.210+0000 7ff7ab439700 1 -- 172.21.15.88:0/108118091 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff7a40f5410 msgr2=0x7ff7a41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:20.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:20.210+0000 7ff7ab439700 1 --2- 172.21.15.88:0/108118091 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff7a40f5410 0x7ff7a41067e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7ff798005010 tx=0x7ff79800b040).stop 2024-02-19T22:12:20.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:20.210+0000 7ff7ab439700 1 -- 172.21.15.88:0/108118091 shutdown_connections 2024-02-19T22:12:20.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:20.210+0000 7ff7ab439700 1 --2- 172.21.15.88:0/108118091 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7ff79002ff10 0x7ff7900323d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:20.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:20.210+0000 7ff7ab439700 1 --2- 172.21.15.88:0/108118091 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff7a40f5410 0x7ff7a41067e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:20.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:20.210+0000 7ff7ab439700 1 -- 172.21.15.88:0/108118091 >> 172.21.15.88:0/108118091 conn(0x7ff7a40f0b70 msgr2=0x7ff7a40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:20.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:20.210+0000 7ff7ab439700 1 -- 172.21.15.88:0/108118091 shutdown_connections 2024-02-19T22:12:20.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:20.210+0000 7ff7ab439700 1 -- 172.21.15.88:0/108118091 wait complete. 2024-02-19T22:12:20.212 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:12:21.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:21 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:12:19.638425+0000 mgr.smithi052.ghqrtk (mgr.14162) 79 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:21.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:21 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:12:20.208989+0000 mon.smithi052 (mon.0) 153 : audit [DBG] from='client.? 172.21.15.88:0/108118091' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:12:21.837 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:12:21.837 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:12:23.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:23 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:12:21.638876+0000 mgr.smithi052.ghqrtk (mgr.14162) 80 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:23.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:23.743+0000 7fee63cbf700 1 -- 172.21.15.88:0/1408235982 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fee5c0f6510 msgr2=0x7fee5c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:23.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:23.743+0000 7fee63cbf700 1 --2- 172.21.15.88:0/1408235982 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fee5c0f6510 0x7fee5c0f6930 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fee4c009fb0 tx=0x7fee4c009b30).stop 2024-02-19T22:12:23.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:23.743+0000 7fee63cbf700 1 -- 172.21.15.88:0/1408235982 shutdown_connections 2024-02-19T22:12:23.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:23.743+0000 7fee63cbf700 1 --2- 172.21.15.88:0/1408235982 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fee5c0f6510 0x7fee5c0f6930 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:23.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:23.743+0000 7fee63cbf700 1 -- 172.21.15.88:0/1408235982 >> 172.21.15.88:0/1408235982 conn(0x7fee5c0f1c70 msgr2=0x7fee5c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:23.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:23.743+0000 7fee63cbf700 1 -- 172.21.15.88:0/1408235982 shutdown_connections 2024-02-19T22:12:23.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:23.743+0000 7fee63cbf700 1 -- 172.21.15.88:0/1408235982 wait complete. 2024-02-19T22:12:23.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:23.744+0000 7fee63cbf700 1 Processor -- start 2024-02-19T22:12:23.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:23.744+0000 7fee63cbf700 1 -- start start 2024-02-19T22:12:23.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:23.744+0000 7fee63cbf700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fee5c0f6510 0x7fee5c108a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:23.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:23.744+0000 7fee63cbf700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee5c108ef0 con 0x7fee5c0f6510 2024-02-19T22:12:23.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:23.744+0000 7fee61a5b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fee5c0f6510 0x7fee5c108a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:23.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:23.744+0000 7fee61a5b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fee5c0f6510 0x7fee5c108a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:52222/0 (socket says 172.21.15.88:52222) 2024-02-19T22:12:23.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:23.744+0000 7fee61a5b700 1 -- 172.21.15.88:0/2469914773 learned_addr learned my addr 172.21.15.88:0/2469914773 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:12:23.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:23.744+0000 7fee61a5b700 1 -- 172.21.15.88:0/2469914773 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee4c005040 con 0x7fee5c0f6510 2024-02-19T22:12:23.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:23.745+0000 7fee61a5b700 1 --2- 172.21.15.88:0/2469914773 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fee5c0f6510 0x7fee5c108a00 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fee4c005010 tx=0x7fee4c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:23.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:23.745+0000 7fee527fc700 1 -- 172.21.15.88:0/2469914773 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee4c009320 con 0x7fee5c0f6510 2024-02-19T22:12:23.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:23.745+0000 7fee63cbf700 1 -- 172.21.15.88:0/2469914773 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee5c1090f0 con 0x7fee5c0f6510 2024-02-19T22:12:23.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:23.745+0000 7fee527fc700 1 -- 172.21.15.88:0/2469914773 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fee4c013380 con 0x7fee5c0f6510 2024-02-19T22:12:23.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:23.745+0000 7fee527fc700 1 -- 172.21.15.88:0/2469914773 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee4c014e50 con 0x7fee5c0f6510 2024-02-19T22:12:23.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:23.745+0000 7fee63cbf700 1 -- 172.21.15.88:0/2469914773 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee5c109580 con 0x7fee5c0f6510 2024-02-19T22:12:23.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:23.745+0000 7fee527fc700 1 -- 172.21.15.88:0/2469914773 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fee4c014420 con 0x7fee5c0f6510 2024-02-19T22:12:23.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:23.746+0000 7fee63cbf700 1 -- 172.21.15.88:0/2469914773 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee5c045ad0 con 0x7fee5c0f6510 2024-02-19T22:12:23.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:23.746+0000 7fee527fc700 1 --2- 172.21.15.88:0/2469914773 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fee4802ff10 0x7fee480323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:23.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:23.746+0000 7fee527fc700 1 -- 172.21.15.88:0/2469914773 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fee4c040e30 con 0x7fee5c0f6510 2024-02-19T22:12:23.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:23.746+0000 7fee6125a700 1 --2- 172.21.15.88:0/2469914773 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fee4802ff10 0x7fee480323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:23.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:23.746+0000 7fee6125a700 1 --2- 172.21.15.88:0/2469914773 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fee4802ff10 0x7fee480323d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fee580096a0 tx=0x7fee58006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:23.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:23.750+0000 7fee527fc700 1 -- 172.21.15.88:0/2469914773 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fee4c0095f0 con 0x7fee5c0f6510 2024-02-19T22:12:23.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:23.955+0000 7fee63cbf700 1 -- 172.21.15.88:0/2469914773 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fee5c10a430 con 0x7fee5c0f6510 2024-02-19T22:12:23.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:23.956+0000 7fee527fc700 1 -- 172.21.15.88:0/2469914773 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fee4c01db70 con 0x7fee5c0f6510 2024-02-19T22:12:23.957 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:12:23.957 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:12:23.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:23.957+0000 7fee63cbf700 1 -- 172.21.15.88:0/2469914773 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fee4802ff10 msgr2=0x7fee480323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:23.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:23.957+0000 7fee63cbf700 1 --2- 172.21.15.88:0/2469914773 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fee4802ff10 0x7fee480323d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fee580096a0 tx=0x7fee58006b40).stop 2024-02-19T22:12:23.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:23.957+0000 7fee63cbf700 1 -- 172.21.15.88:0/2469914773 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fee5c0f6510 msgr2=0x7fee5c108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:23.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:23.957+0000 7fee63cbf700 1 --2- 172.21.15.88:0/2469914773 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fee5c0f6510 0x7fee5c108a00 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fee4c005010 tx=0x7fee4c00b040).stop 2024-02-19T22:12:23.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:23.957+0000 7fee63cbf700 1 -- 172.21.15.88:0/2469914773 shutdown_connections 2024-02-19T22:12:23.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:23.957+0000 7fee63cbf700 1 --2- 172.21.15.88:0/2469914773 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fee4802ff10 0x7fee480323d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:23.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:23.957+0000 7fee63cbf700 1 --2- 172.21.15.88:0/2469914773 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fee5c0f6510 0x7fee5c108a00 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:23.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:23.957+0000 7fee63cbf700 1 -- 172.21.15.88:0/2469914773 >> 172.21.15.88:0/2469914773 conn(0x7fee5c0f1c70 msgr2=0x7fee5c0f26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:23.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:23.957+0000 7fee63cbf700 1 -- 172.21.15.88:0/2469914773 shutdown_connections 2024-02-19T22:12:23.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:23.957+0000 7fee63cbf700 1 -- 172.21.15.88:0/2469914773 wait complete. 2024-02-19T22:12:23.959 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:12:24.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:24 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:12:23.956056+0000 mon.smithi052 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.88:0/2469914773' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:12:25.557 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:12:25.557 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:12:25.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:25 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:12:23.639213+0000 mgr.smithi052.ghqrtk (mgr.14162) 81 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:27.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:27.552+0000 7f5136d5f700 1 -- 172.21.15.88:0/3116008149 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f51300f62a0 msgr2=0x7f51300f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:27.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:27.552+0000 7f5136d5f700 1 --2- 172.21.15.88:0/3116008149 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f51300f62a0 0x7f51300f66c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f512c009fb0 tx=0x7f512c009b30).stop 2024-02-19T22:12:27.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:27.552+0000 7f5136d5f700 1 -- 172.21.15.88:0/3116008149 shutdown_connections 2024-02-19T22:12:27.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:27.552+0000 7f5136d5f700 1 --2- 172.21.15.88:0/3116008149 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f51300f62a0 0x7f51300f66c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:27.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:27.552+0000 7f5136d5f700 1 -- 172.21.15.88:0/3116008149 >> 172.21.15.88:0/3116008149 conn(0x7f51300f1aa0 msgr2=0x7f51300f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:27.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:27.552+0000 7f5136d5f700 1 -- 172.21.15.88:0/3116008149 shutdown_connections 2024-02-19T22:12:27.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:27.552+0000 7f5136d5f700 1 -- 172.21.15.88:0/3116008149 wait complete. 2024-02-19T22:12:27.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:27.552+0000 7f5136d5f700 1 Processor -- start 2024-02-19T22:12:27.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:27.552+0000 7f5136d5f700 1 -- start start 2024-02-19T22:12:27.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:27.553+0000 7f5136d5f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f51300f62a0 0x7f513006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:27.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:27.553+0000 7f5136d5f700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5130069e20 con 0x7f51300f62a0 2024-02-19T22:12:27.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:27.553+0000 7f5135d5d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f51300f62a0 0x7f513006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:27.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:27.553+0000 7f5135d5d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f51300f62a0 0x7f513006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:52230/0 (socket says 172.21.15.88:52230) 2024-02-19T22:12:27.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:27.553+0000 7f5135d5d700 1 -- 172.21.15.88:0/2310339281 learned_addr learned my addr 172.21.15.88:0/2310339281 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:12:27.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:27.553+0000 7f5135d5d700 1 -- 172.21.15.88:0/2310339281 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f512c005040 con 0x7f51300f62a0 2024-02-19T22:12:27.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:27.553+0000 7f5135d5d700 1 --2- 172.21.15.88:0/2310339281 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f51300f62a0 0x7f513006b9a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f512c0089f0 tx=0x7f512c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:27.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:27.554+0000 7f51267fc700 1 -- 172.21.15.88:0/2310339281 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f512c009320 con 0x7f51300f62a0 2024-02-19T22:12:27.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:27.554+0000 7f5136d5f700 1 -- 172.21.15.88:0/2310339281 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5130069fc0 con 0x7f51300f62a0 2024-02-19T22:12:27.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:27.554+0000 7f51267fc700 1 -- 172.21.15.88:0/2310339281 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f512c013380 con 0x7f51300f62a0 2024-02-19T22:12:27.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:27.554+0000 7f51267fc700 1 -- 172.21.15.88:0/2310339281 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f512c014e20 con 0x7f51300f62a0 2024-02-19T22:12:27.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:27.554+0000 7f5136d5f700 1 -- 172.21.15.88:0/2310339281 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f513006a450 con 0x7f51300f62a0 2024-02-19T22:12:27.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:27.554+0000 7f51267fc700 1 -- 172.21.15.88:0/2310339281 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f512c014420 con 0x7f51300f62a0 2024-02-19T22:12:27.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:27.554+0000 7f5136d5f700 1 -- 172.21.15.88:0/2310339281 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5130059100 con 0x7f51300f62a0 2024-02-19T22:12:27.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:27.555+0000 7f51267fc700 1 --2- 172.21.15.88:0/2310339281 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f511c02ff10 0x7f511c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:27.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:27.555+0000 7f51267fc700 1 -- 172.21.15.88:0/2310339281 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f512c040ef0 con 0x7f51300f62a0 2024-02-19T22:12:27.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:27.555+0000 7f513555c700 1 --2- 172.21.15.88:0/2310339281 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f511c02ff10 0x7f511c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:27.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:27.555+0000 7f513555c700 1 --2- 172.21.15.88:0/2310339281 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f511c02ff10 0x7f511c0323d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f51200096a0 tx=0x7f5120006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:27.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:27.559+0000 7f51267fc700 1 -- 172.21.15.88:0/2310339281 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f512c014790 con 0x7f51300f62a0 2024-02-19T22:12:27.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:27 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:12:25.639547+0000 mgr.smithi052.ghqrtk (mgr.14162) 82 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:27.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:27.763+0000 7f5136d5f700 1 -- 172.21.15.88:0/2310339281 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5130045ad0 con 0x7f51300f62a0 2024-02-19T22:12:27.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:27.763+0000 7f51267fc700 1 -- 172.21.15.88:0/2310339281 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f512c01d7f0 con 0x7f51300f62a0 2024-02-19T22:12:27.764 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:12:27.764 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:12:27.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:27.765+0000 7f5136d5f700 1 -- 172.21.15.88:0/2310339281 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f511c02ff10 msgr2=0x7f511c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:27.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:27.765+0000 7f5136d5f700 1 --2- 172.21.15.88:0/2310339281 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f511c02ff10 0x7f511c0323d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f51200096a0 tx=0x7f5120006b40).stop 2024-02-19T22:12:27.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:27.765+0000 7f5136d5f700 1 -- 172.21.15.88:0/2310339281 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f51300f62a0 msgr2=0x7f513006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:27.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:27.765+0000 7f5136d5f700 1 --2- 172.21.15.88:0/2310339281 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f51300f62a0 0x7f513006b9a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f512c0089f0 tx=0x7f512c00b040).stop 2024-02-19T22:12:27.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:27.765+0000 7f5136d5f700 1 -- 172.21.15.88:0/2310339281 shutdown_connections 2024-02-19T22:12:27.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:27.765+0000 7f5136d5f700 1 --2- 172.21.15.88:0/2310339281 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f511c02ff10 0x7f511c0323d0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:27.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:27.765+0000 7f5136d5f700 1 --2- 172.21.15.88:0/2310339281 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f51300f62a0 0x7f513006b9a0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:27.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:27.765+0000 7f5136d5f700 1 -- 172.21.15.88:0/2310339281 >> 172.21.15.88:0/2310339281 conn(0x7f51300f1aa0 msgr2=0x7f51300f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:27.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:27.765+0000 7f5136d5f700 1 -- 172.21.15.88:0/2310339281 shutdown_connections 2024-02-19T22:12:27.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:27.765+0000 7f5136d5f700 1 -- 172.21.15.88:0/2310339281 wait complete. 2024-02-19T22:12:27.767 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:12:28.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:28 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:12:27.763630+0000 mon.smithi052 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.88:0/2310339281' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:12:29.416 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:12:29.416 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:12:29.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:29 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:12:27.639843+0000 mgr.smithi052.ghqrtk (mgr.14162) 83 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:31.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:31.156+0000 7f04e2aec700 1 -- 172.21.15.88:0/3431634086 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f04dc0f6510 msgr2=0x7f04dc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:31.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:31.156+0000 7f04e2aec700 1 --2- 172.21.15.88:0/3431634086 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f04dc0f6510 0x7f04dc0f6930 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f04d0009fb0 tx=0x7f04d0009b30).stop 2024-02-19T22:12:31.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:31.156+0000 7f04e2aec700 1 -- 172.21.15.88:0/3431634086 shutdown_connections 2024-02-19T22:12:31.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:31.156+0000 7f04e2aec700 1 --2- 172.21.15.88:0/3431634086 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f04dc0f6510 0x7f04dc0f6930 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:31.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:31.156+0000 7f04e2aec700 1 -- 172.21.15.88:0/3431634086 >> 172.21.15.88:0/3431634086 conn(0x7f04dc0f1c70 msgr2=0x7f04dc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:31.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:31.156+0000 7f04e2aec700 1 -- 172.21.15.88:0/3431634086 shutdown_connections 2024-02-19T22:12:31.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:31.156+0000 7f04e2aec700 1 -- 172.21.15.88:0/3431634086 wait complete. 2024-02-19T22:12:31.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:31.156+0000 7f04e2aec700 1 Processor -- start 2024-02-19T22:12:31.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:31.156+0000 7f04e2aec700 1 -- start start 2024-02-19T22:12:31.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:31.157+0000 7f04e2aec700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f04dc0f6510 0x7f04dc06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:31.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:31.157+0000 7f04e2aec700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04dc069e20 con 0x7f04dc0f6510 2024-02-19T22:12:31.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:31.157+0000 7f04e0888700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f04dc0f6510 0x7f04dc06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:31.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:31.157+0000 7f04e0888700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f04dc0f6510 0x7f04dc06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:52252/0 (socket says 172.21.15.88:52252) 2024-02-19T22:12:31.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:31.157+0000 7f04e0888700 1 -- 172.21.15.88:0/1971929239 learned_addr learned my addr 172.21.15.88:0/1971929239 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:12:31.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:31.157+0000 7f04e0888700 1 -- 172.21.15.88:0/1971929239 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04d0005040 con 0x7f04dc0f6510 2024-02-19T22:12:31.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:31.157+0000 7f04e0888700 1 --2- 172.21.15.88:0/1971929239 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f04dc0f6510 0x7f04dc06b9a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f04d0005010 tx=0x7f04d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:31.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:31.158+0000 7f04d97fa700 1 -- 172.21.15.88:0/1971929239 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f04d0009320 con 0x7f04dc0f6510 2024-02-19T22:12:31.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:31.158+0000 7f04d97fa700 1 -- 172.21.15.88:0/1971929239 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f04d0013380 con 0x7f04dc0f6510 2024-02-19T22:12:31.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:31.158+0000 7f04e2aec700 1 -- 172.21.15.88:0/1971929239 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04dc069fc0 con 0x7f04dc0f6510 2024-02-19T22:12:31.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:31.158+0000 7f04d97fa700 1 -- 172.21.15.88:0/1971929239 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f04d0014e50 con 0x7f04dc0f6510 2024-02-19T22:12:31.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:31.158+0000 7f04e2aec700 1 -- 172.21.15.88:0/1971929239 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04dc06a450 con 0x7f04dc0f6510 2024-02-19T22:12:31.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:31.158+0000 7f04d97fa700 1 -- 172.21.15.88:0/1971929239 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f04d00095f0 con 0x7f04dc0f6510 2024-02-19T22:12:31.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:31.159+0000 7f04e2aec700 1 -- 172.21.15.88:0/1971929239 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04dc059100 con 0x7f04dc0f6510 2024-02-19T22:12:31.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:31.159+0000 7f04d97fa700 1 --2- 172.21.15.88:0/1971929239 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f04cc02ff10 0x7f04cc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:31.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:31.159+0000 7f04d97fa700 1 -- 172.21.15.88:0/1971929239 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f04d0040f10 con 0x7f04dc0f6510 2024-02-19T22:12:31.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:31.159+0000 7f04dbfff700 1 --2- 172.21.15.88:0/1971929239 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f04cc02ff10 0x7f04cc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:31.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:31.159+0000 7f04dbfff700 1 --2- 172.21.15.88:0/1971929239 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f04cc02ff10 0x7f04cc0323d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f04c80096a0 tx=0x7f04c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:31.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:31.164+0000 7f04d97fa700 1 -- 172.21.15.88:0/1971929239 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f04d0014420 con 0x7f04dc0f6510 2024-02-19T22:12:31.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:31.369+0000 7f04e2aec700 1 -- 172.21.15.88:0/1971929239 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f04dc0fc0a0 con 0x7f04dc0f6510 2024-02-19T22:12:31.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:31.370+0000 7f04d97fa700 1 -- 172.21.15.88:0/1971929239 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f04d0010230 con 0x7f04dc0f6510 2024-02-19T22:12:31.370 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:12:31.370 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:12:31.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:31.371+0000 7f04e2aec700 1 -- 172.21.15.88:0/1971929239 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f04cc02ff10 msgr2=0x7f04cc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:31.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:31.371+0000 7f04e2aec700 1 --2- 172.21.15.88:0/1971929239 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f04cc02ff10 0x7f04cc0323d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f04c80096a0 tx=0x7f04c8006b40).stop 2024-02-19T22:12:31.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:31.371+0000 7f04e2aec700 1 -- 172.21.15.88:0/1971929239 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f04dc0f6510 msgr2=0x7f04dc06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:31.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:31.371+0000 7f04e2aec700 1 --2- 172.21.15.88:0/1971929239 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f04dc0f6510 0x7f04dc06b9a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f04d0005010 tx=0x7f04d000b040).stop 2024-02-19T22:12:31.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:31.371+0000 7f04e2aec700 1 -- 172.21.15.88:0/1971929239 shutdown_connections 2024-02-19T22:12:31.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:31.371+0000 7f04e2aec700 1 --2- 172.21.15.88:0/1971929239 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f04cc02ff10 0x7f04cc0323d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:31.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:31.371+0000 7f04e2aec700 1 --2- 172.21.15.88:0/1971929239 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f04dc0f6510 0x7f04dc06b9a0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:31.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:31.371+0000 7f04e2aec700 1 -- 172.21.15.88:0/1971929239 >> 172.21.15.88:0/1971929239 conn(0x7f04dc0f1c70 msgr2=0x7f04dc0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:31.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:31.371+0000 7f04e2aec700 1 -- 172.21.15.88:0/1971929239 shutdown_connections 2024-02-19T22:12:31.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:31.371+0000 7f04e2aec700 1 -- 172.21.15.88:0/1971929239 wait complete. 2024-02-19T22:12:31.373 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:12:31.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:31 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:12:29.640297+0000 mgr.smithi052.ghqrtk (mgr.14162) 84 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:32.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:32 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:12:31.370085+0000 mon.smithi052 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.88:0/1971929239' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:12:33.044 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:12:33.044 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:12:33.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:33 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:12:31.640672+0000 mgr.smithi052.ghqrtk (mgr.14162) 85 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:34.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:34.880+0000 7f3c6fb79700 1 -- 172.21.15.88:0/580235172 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c680f6510 msgr2=0x7f3c680f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:34.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:34.880+0000 7f3c6fb79700 1 --2- 172.21.15.88:0/580235172 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c680f6510 0x7f3c680f6930 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f3c5c009fb0 tx=0x7f3c5c009b30).stop 2024-02-19T22:12:34.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:34.880+0000 7f3c6fb79700 1 -- 172.21.15.88:0/580235172 shutdown_connections 2024-02-19T22:12:34.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:34.881+0000 7f3c6fb79700 1 --2- 172.21.15.88:0/580235172 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c680f6510 0x7f3c680f6930 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:34.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:34.881+0000 7f3c6fb79700 1 -- 172.21.15.88:0/580235172 >> 172.21.15.88:0/580235172 conn(0x7f3c680f1c70 msgr2=0x7f3c680f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:34.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:34.881+0000 7f3c6fb79700 1 -- 172.21.15.88:0/580235172 shutdown_connections 2024-02-19T22:12:34.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:34.881+0000 7f3c6fb79700 1 -- 172.21.15.88:0/580235172 wait complete. 2024-02-19T22:12:34.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:34.881+0000 7f3c6fb79700 1 Processor -- start 2024-02-19T22:12:34.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:34.881+0000 7f3c6fb79700 1 -- start start 2024-02-19T22:12:34.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:34.882+0000 7f3c6fb79700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c680f6510 0x7f3c681067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:34.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:34.882+0000 7f3c6fb79700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c68106cd0 con 0x7f3c680f6510 2024-02-19T22:12:34.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:34.882+0000 7f3c6d915700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c680f6510 0x7f3c681067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:34.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:34.882+0000 7f3c6d915700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c680f6510 0x7f3c681067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:59198/0 (socket says 172.21.15.88:59198) 2024-02-19T22:12:34.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:34.882+0000 7f3c6d915700 1 -- 172.21.15.88:0/3289724413 learned_addr learned my addr 172.21.15.88:0/3289724413 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:12:34.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:34.882+0000 7f3c6d915700 1 -- 172.21.15.88:0/3289724413 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c5c005040 con 0x7f3c680f6510 2024-02-19T22:12:34.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:34.882+0000 7f3c6d915700 1 --2- 172.21.15.88:0/3289724413 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c680f6510 0x7f3c681067e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f3c5c005010 tx=0x7f3c5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:34.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:34.882+0000 7f3c5a7fc700 1 -- 172.21.15.88:0/3289724413 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c5c009320 con 0x7f3c680f6510 2024-02-19T22:12:34.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:34.882+0000 7f3c6fb79700 1 -- 172.21.15.88:0/3289724413 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c68106ed0 con 0x7f3c680f6510 2024-02-19T22:12:34.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:34.882+0000 7f3c5a7fc700 1 -- 172.21.15.88:0/3289724413 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3c5c013380 con 0x7f3c680f6510 2024-02-19T22:12:34.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:34.882+0000 7f3c5a7fc700 1 -- 172.21.15.88:0/3289724413 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c5c014e50 con 0x7f3c680f6510 2024-02-19T22:12:34.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:34.882+0000 7f3c6fb79700 1 -- 172.21.15.88:0/3289724413 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c68107360 con 0x7f3c680f6510 2024-02-19T22:12:34.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:34.883+0000 7f3c5a7fc700 1 -- 172.21.15.88:0/3289724413 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3c5c0095f0 con 0x7f3c680f6510 2024-02-19T22:12:34.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:34.883+0000 7f3c6fb79700 1 -- 172.21.15.88:0/3289724413 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c68045ad0 con 0x7f3c680f6510 2024-02-19T22:12:34.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:34.883+0000 7f3c5a7fc700 1 --2- 172.21.15.88:0/3289724413 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f3c54034320 0x7f3c540367e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:34.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:34.883+0000 7f3c5a7fc700 1 -- 172.21.15.88:0/3289724413 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3c5c040fa0 con 0x7f3c680f6510 2024-02-19T22:12:34.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:34.883+0000 7f3c6d114700 1 --2- 172.21.15.88:0/3289724413 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f3c54034320 0x7f3c540367e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:34.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:34.884+0000 7f3c6d114700 1 --2- 172.21.15.88:0/3289724413 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f3c54034320 0x7f3c540367e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f3c640096a0 tx=0x7f3c64006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:34.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:34.888+0000 7f3c5a7fc700 1 -- 172.21.15.88:0/3289724413 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3c5c014420 con 0x7f3c680f6510 2024-02-19T22:12:35.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:35.095+0000 7f3c6fb79700 1 -- 172.21.15.88:0/3289724413 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3c68108210 con 0x7f3c680f6510 2024-02-19T22:12:35.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:35.096+0000 7f3c5a7fc700 1 -- 172.21.15.88:0/3289724413 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3c5c010230 con 0x7f3c680f6510 2024-02-19T22:12:35.098 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:12:35.098 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:12:35.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:35.097+0000 7f3c6fb79700 1 -- 172.21.15.88:0/3289724413 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f3c54034320 msgr2=0x7f3c540367e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:35.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:35.097+0000 7f3c6fb79700 1 --2- 172.21.15.88:0/3289724413 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f3c54034320 0x7f3c540367e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f3c640096a0 tx=0x7f3c64006b40).stop 2024-02-19T22:12:35.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:35.097+0000 7f3c6fb79700 1 -- 172.21.15.88:0/3289724413 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c680f6510 msgr2=0x7f3c681067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:35.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:35.097+0000 7f3c6fb79700 1 --2- 172.21.15.88:0/3289724413 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c680f6510 0x7f3c681067e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f3c5c005010 tx=0x7f3c5c00b040).stop 2024-02-19T22:12:35.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:35.098+0000 7f3c6fb79700 1 -- 172.21.15.88:0/3289724413 shutdown_connections 2024-02-19T22:12:35.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:35.098+0000 7f3c6fb79700 1 --2- 172.21.15.88:0/3289724413 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f3c54034320 0x7f3c540367e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:35.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:35.098+0000 7f3c6fb79700 1 --2- 172.21.15.88:0/3289724413 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c680f6510 0x7f3c681067e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:35.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:35.098+0000 7f3c6fb79700 1 -- 172.21.15.88:0/3289724413 >> 172.21.15.88:0/3289724413 conn(0x7f3c680f1c70 msgr2=0x7f3c680f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:35.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:35.098+0000 7f3c6fb79700 1 -- 172.21.15.88:0/3289724413 shutdown_connections 2024-02-19T22:12:35.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:35.098+0000 7f3c6fb79700 1 -- 172.21.15.88:0/3289724413 wait complete. 2024-02-19T22:12:35.100 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:12:35.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:35 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:12:33.641094+0000 mgr.smithi052.ghqrtk (mgr.14162) 86 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:35.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:35 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:12:35.097492+0000 mon.smithi052 (mon.0) 157 : audit [DBG] from='client.? 172.21.15.88:0/3289724413' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:12:36.673 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:12:36.673 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:12:37.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:37 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:12:35.641449+0000 mgr.smithi052.ghqrtk (mgr.14162) 87 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:38.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:38.450+0000 7feffb0d3700 1 -- 172.21.15.88:0/2000833826 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feff40f6510 msgr2=0x7feff40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:38.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:38.450+0000 7feffb0d3700 1 --2- 172.21.15.88:0/2000833826 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feff40f6510 0x7feff40f6930 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fefe8009fb0 tx=0x7fefe8009b30).stop 2024-02-19T22:12:38.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:38.450+0000 7feffb0d3700 1 -- 172.21.15.88:0/2000833826 shutdown_connections 2024-02-19T22:12:38.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:38.450+0000 7feffb0d3700 1 --2- 172.21.15.88:0/2000833826 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feff40f6510 0x7feff40f6930 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:38.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:38.450+0000 7feffb0d3700 1 -- 172.21.15.88:0/2000833826 >> 172.21.15.88:0/2000833826 conn(0x7feff40f1c70 msgr2=0x7feff40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:38.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:38.450+0000 7feffb0d3700 1 -- 172.21.15.88:0/2000833826 shutdown_connections 2024-02-19T22:12:38.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:38.450+0000 7feffb0d3700 1 -- 172.21.15.88:0/2000833826 wait complete. 2024-02-19T22:12:38.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:38.450+0000 7feffb0d3700 1 Processor -- start 2024-02-19T22:12:38.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:38.450+0000 7feffb0d3700 1 -- start start 2024-02-19T22:12:38.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:38.451+0000 7feffb0d3700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feff40f6510 0x7feff41067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:38.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:38.451+0000 7feffb0d3700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff4106cd0 con 0x7feff40f6510 2024-02-19T22:12:38.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:38.451+0000 7feff8e6f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feff40f6510 0x7feff41067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:38.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:38.451+0000 7feff8e6f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feff40f6510 0x7feff41067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:59214/0 (socket says 172.21.15.88:59214) 2024-02-19T22:12:38.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:38.451+0000 7feff8e6f700 1 -- 172.21.15.88:0/3096225835 learned_addr learned my addr 172.21.15.88:0/3096225835 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:12:38.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:38.451+0000 7feff8e6f700 1 -- 172.21.15.88:0/3096225835 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe8005040 con 0x7feff40f6510 2024-02-19T22:12:38.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:38.451+0000 7feff8e6f700 1 --2- 172.21.15.88:0/3096225835 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feff40f6510 0x7feff41067e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fefe8005010 tx=0x7fefe800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:38.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:38.451+0000 7feff17fa700 1 -- 172.21.15.88:0/3096225835 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefe8009320 con 0x7feff40f6510 2024-02-19T22:12:38.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:38.452+0000 7feff17fa700 1 -- 172.21.15.88:0/3096225835 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fefe8013380 con 0x7feff40f6510 2024-02-19T22:12:38.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:38.451+0000 7feffb0d3700 1 -- 172.21.15.88:0/3096225835 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feff4106ed0 con 0x7feff40f6510 2024-02-19T22:12:38.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:38.452+0000 7feff17fa700 1 -- 172.21.15.88:0/3096225835 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefe8014e50 con 0x7feff40f6510 2024-02-19T22:12:38.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:38.452+0000 7feffb0d3700 1 -- 172.21.15.88:0/3096225835 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feff4107360 con 0x7feff40f6510 2024-02-19T22:12:38.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:38.452+0000 7feff17fa700 1 -- 172.21.15.88:0/3096225835 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fefe8014420 con 0x7feff40f6510 2024-02-19T22:12:38.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:38.452+0000 7feff17fa700 1 --2- 172.21.15.88:0/3096225835 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fefe402ff10 0x7fefe40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:38.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:38.452+0000 7feffb0d3700 1 -- 172.21.15.88:0/3096225835 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feff4045ad0 con 0x7feff40f6510 2024-02-19T22:12:38.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:38.453+0000 7feff17fa700 1 -- 172.21.15.88:0/3096225835 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fefe8040e30 con 0x7feff40f6510 2024-02-19T22:12:38.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:38.453+0000 7feff3fff700 1 --2- 172.21.15.88:0/3096225835 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fefe402ff10 0x7fefe40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:38.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:38.453+0000 7feff3fff700 1 --2- 172.21.15.88:0/3096225835 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fefe402ff10 0x7fefe40323d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fefe00096a0 tx=0x7fefe0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:38.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:38.457+0000 7feff17fa700 1 -- 172.21.15.88:0/3096225835 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fefe801db70 con 0x7feff40f6510 2024-02-19T22:12:38.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:38.663+0000 7feffb0d3700 1 -- 172.21.15.88:0/3096225835 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feff4108210 con 0x7feff40f6510 2024-02-19T22:12:38.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:38.663+0000 7feff17fa700 1 -- 172.21.15.88:0/3096225835 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fefe8023250 con 0x7feff40f6510 2024-02-19T22:12:38.666 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:12:38.666 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:12:38.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:38.664+0000 7feffb0d3700 1 -- 172.21.15.88:0/3096225835 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fefe402ff10 msgr2=0x7fefe40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:38.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:38.665+0000 7feffb0d3700 1 --2- 172.21.15.88:0/3096225835 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fefe402ff10 0x7fefe40323d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fefe00096a0 tx=0x7fefe0006b40).stop 2024-02-19T22:12:38.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:38.665+0000 7feffb0d3700 1 -- 172.21.15.88:0/3096225835 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feff40f6510 msgr2=0x7feff41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:38.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:38.665+0000 7feffb0d3700 1 --2- 172.21.15.88:0/3096225835 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feff40f6510 0x7feff41067e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fefe8005010 tx=0x7fefe800b040).stop 2024-02-19T22:12:38.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:38.665+0000 7feffb0d3700 1 -- 172.21.15.88:0/3096225835 shutdown_connections 2024-02-19T22:12:38.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:38.665+0000 7feffb0d3700 1 --2- 172.21.15.88:0/3096225835 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fefe402ff10 0x7fefe40323d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:38.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:38.665+0000 7feffb0d3700 1 --2- 172.21.15.88:0/3096225835 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feff40f6510 0x7feff41067e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:38.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:38.665+0000 7feffb0d3700 1 -- 172.21.15.88:0/3096225835 >> 172.21.15.88:0/3096225835 conn(0x7feff40f1c70 msgr2=0x7feff40f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:38.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:38.665+0000 7feffb0d3700 1 -- 172.21.15.88:0/3096225835 shutdown_connections 2024-02-19T22:12:38.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:38.665+0000 7feffb0d3700 1 -- 172.21.15.88:0/3096225835 wait complete. 2024-02-19T22:12:38.667 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:12:39.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:12:37.641874+0000 mgr.smithi052.ghqrtk (mgr.14162) 88 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:39.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:12:38.664617+0000 mon.smithi052 (mon.0) 158 : audit [DBG] from='client.? 172.21.15.88:0/3096225835' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:12:40.264 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:12:40.264 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:12:41.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:41 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:12:39.642343+0000 mgr.smithi052.ghqrtk (mgr.14162) 89 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:42.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:42.094+0000 7f4272f72700 1 -- 172.21.15.88:0/1282083303 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f426c0f5410 msgr2=0x7f426c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:42.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:42.094+0000 7f4272f72700 1 --2- 172.21.15.88:0/1282083303 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f426c0f5410 0x7f426c0f5830 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f425c009fb0 tx=0x7f425c009b30).stop 2024-02-19T22:12:42.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:42.094+0000 7f4272f72700 1 -- 172.21.15.88:0/1282083303 shutdown_connections 2024-02-19T22:12:42.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:42.094+0000 7f4272f72700 1 --2- 172.21.15.88:0/1282083303 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f426c0f5410 0x7f426c0f5830 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:42.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:42.094+0000 7f4272f72700 1 -- 172.21.15.88:0/1282083303 >> 172.21.15.88:0/1282083303 conn(0x7f426c0f0b70 msgr2=0x7f426c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:42.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:42.094+0000 7f4272f72700 1 -- 172.21.15.88:0/1282083303 shutdown_connections 2024-02-19T22:12:42.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:42.094+0000 7f4272f72700 1 -- 172.21.15.88:0/1282083303 wait complete. 2024-02-19T22:12:42.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:42.095+0000 7f4272f72700 1 Processor -- start 2024-02-19T22:12:42.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:42.095+0000 7f4272f72700 1 -- start start 2024-02-19T22:12:42.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:42.095+0000 7f4272f72700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f426c0f5410 0x7f426c1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:42.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:42.095+0000 7f4272f72700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f426c105bd0 con 0x7f426c0f5410 2024-02-19T22:12:42.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:42.096+0000 7f4270d0e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f426c0f5410 0x7f426c1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:42.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:42.096+0000 7f4270d0e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f426c0f5410 0x7f426c1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:59232/0 (socket says 172.21.15.88:59232) 2024-02-19T22:12:42.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:42.096+0000 7f4270d0e700 1 -- 172.21.15.88:0/1429607579 learned_addr learned my addr 172.21.15.88:0/1429607579 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:12:42.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:42.096+0000 7f4270d0e700 1 -- 172.21.15.88:0/1429607579 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f425c005040 con 0x7f426c0f5410 2024-02-19T22:12:42.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:42.096+0000 7f4270d0e700 1 --2- 172.21.15.88:0/1429607579 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f426c0f5410 0x7f426c1056e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f425c005010 tx=0x7f425c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:42.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:42.096+0000 7f42697fa700 1 -- 172.21.15.88:0/1429607579 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f425c009320 con 0x7f426c0f5410 2024-02-19T22:12:42.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:42.096+0000 7f42697fa700 1 -- 172.21.15.88:0/1429607579 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f425c013380 con 0x7f426c0f5410 2024-02-19T22:12:42.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:42.096+0000 7f4272f72700 1 -- 172.21.15.88:0/1429607579 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f426c105dd0 con 0x7f426c0f5410 2024-02-19T22:12:42.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:42.096+0000 7f42697fa700 1 -- 172.21.15.88:0/1429607579 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f425c014e50 con 0x7f426c0f5410 2024-02-19T22:12:42.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:42.096+0000 7f4272f72700 1 -- 172.21.15.88:0/1429607579 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f426c106260 con 0x7f426c0f5410 2024-02-19T22:12:42.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:42.097+0000 7f42697fa700 1 -- 172.21.15.88:0/1429607579 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f425c0263a0 con 0x7f426c0f5410 2024-02-19T22:12:42.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:42.097+0000 7f42697fa700 1 --2- 172.21.15.88:0/1429607579 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f425402ff10 0x7f42540323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:42.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:42.097+0000 7f42697fa700 1 -- 172.21.15.88:0/1429607579 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f425c040f80 con 0x7f426c0f5410 2024-02-19T22:12:42.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:42.097+0000 7f4272f72700 1 -- 172.21.15.88:0/1429607579 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f426c045ad0 con 0x7f426c0f5410 2024-02-19T22:12:42.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:42.097+0000 7f426bfff700 1 --2- 172.21.15.88:0/1429607579 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f425402ff10 0x7f42540323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:42.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:42.098+0000 7f426bfff700 1 --2- 172.21.15.88:0/1429607579 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f425402ff10 0x7f42540323d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f42600096a0 tx=0x7f4260006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:42.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:42.102+0000 7f42697fa700 1 -- 172.21.15.88:0/1429607579 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f425c012920 con 0x7f426c0f5410 2024-02-19T22:12:42.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:42.307+0000 7f4272f72700 1 -- 172.21.15.88:0/1429607579 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f426c106f80 con 0x7f426c0f5410 2024-02-19T22:12:42.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:42.307+0000 7f42697fa700 1 -- 172.21.15.88:0/1429607579 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f425c02b020 con 0x7f426c0f5410 2024-02-19T22:12:42.309 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:12:42.310 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:12:42.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:42.309+0000 7f4272f72700 1 -- 172.21.15.88:0/1429607579 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f425402ff10 msgr2=0x7f42540323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:42.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:42.309+0000 7f4272f72700 1 --2- 172.21.15.88:0/1429607579 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f425402ff10 0x7f42540323d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f42600096a0 tx=0x7f4260006b40).stop 2024-02-19T22:12:42.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:42.309+0000 7f4272f72700 1 -- 172.21.15.88:0/1429607579 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f426c0f5410 msgr2=0x7f426c1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:42.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:42.309+0000 7f4272f72700 1 --2- 172.21.15.88:0/1429607579 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f426c0f5410 0x7f426c1056e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f425c005010 tx=0x7f425c00b040).stop 2024-02-19T22:12:42.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:42.309+0000 7f4272f72700 1 -- 172.21.15.88:0/1429607579 shutdown_connections 2024-02-19T22:12:42.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:42.309+0000 7f4272f72700 1 --2- 172.21.15.88:0/1429607579 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f425402ff10 0x7f42540323d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:42.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:42.309+0000 7f4272f72700 1 --2- 172.21.15.88:0/1429607579 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f426c0f5410 0x7f426c1056e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:42.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:42.309+0000 7f4272f72700 1 -- 172.21.15.88:0/1429607579 >> 172.21.15.88:0/1429607579 conn(0x7f426c0f0b70 msgr2=0x7f426c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:42.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:42.309+0000 7f4272f72700 1 -- 172.21.15.88:0/1429607579 shutdown_connections 2024-02-19T22:12:42.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:42.309+0000 7f4272f72700 1 -- 172.21.15.88:0/1429607579 wait complete. 2024-02-19T22:12:42.312 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:12:43.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:43 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:12:41.642662+0000 mgr.smithi052.ghqrtk (mgr.14162) 90 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:43.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:43 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:12:42.308903+0000 mon.smithi052 (mon.0) 159 : audit [DBG] from='client.? 172.21.15.88:0/1429607579' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:12:43.925 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:12:43.925 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:12:44.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:44 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:12:43.747160+0000 mon.smithi052 (mon.0) 160 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi052.ghqrtk/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:12:44.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:44 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:12:43.750635+0000 mon.smithi052 (mon.0) 161 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi052.ghqrtk/trash_purge_schedule"}]: dispatch 2024-02-19T22:12:45.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:45 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:12:43.642981+0000 mgr.smithi052.ghqrtk (mgr.14162) 91 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:45.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:45.654+0000 7f6e9f350700 1 -- 172.21.15.88:0/171439412 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6e980f5410 msgr2=0x7f6e980f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:45.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:45.654+0000 7f6e9f350700 1 --2- 172.21.15.88:0/171439412 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6e980f5410 0x7f6e980f5830 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f6e88009fb0 tx=0x7f6e88009b30).stop 2024-02-19T22:12:45.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:45.654+0000 7f6e9f350700 1 -- 172.21.15.88:0/171439412 shutdown_connections 2024-02-19T22:12:45.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:45.654+0000 7f6e9f350700 1 --2- 172.21.15.88:0/171439412 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6e980f5410 0x7f6e980f5830 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:45.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:45.654+0000 7f6e9f350700 1 -- 172.21.15.88:0/171439412 >> 172.21.15.88:0/171439412 conn(0x7f6e980f0b70 msgr2=0x7f6e980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:45.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:45.654+0000 7f6e9f350700 1 -- 172.21.15.88:0/171439412 shutdown_connections 2024-02-19T22:12:45.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:45.654+0000 7f6e9f350700 1 -- 172.21.15.88:0/171439412 wait complete. 2024-02-19T22:12:45.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:45.655+0000 7f6e9f350700 1 Processor -- start 2024-02-19T22:12:45.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:45.655+0000 7f6e9f350700 1 -- start start 2024-02-19T22:12:45.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:45.655+0000 7f6e9f350700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6e980f5410 0x7f6e981056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:45.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:45.655+0000 7f6e9f350700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e98105bd0 con 0x7f6e980f5410 2024-02-19T22:12:45.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:45.655+0000 7f6e9d0ec700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6e980f5410 0x7f6e981056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:45.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:45.655+0000 7f6e9d0ec700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6e980f5410 0x7f6e981056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:46934/0 (socket says 172.21.15.88:46934) 2024-02-19T22:12:45.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:45.655+0000 7f6e9d0ec700 1 -- 172.21.15.88:0/1260861363 learned_addr learned my addr 172.21.15.88:0/1260861363 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:12:45.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:45.655+0000 7f6e9d0ec700 1 -- 172.21.15.88:0/1260861363 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e88005040 con 0x7f6e980f5410 2024-02-19T22:12:45.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:45.656+0000 7f6e9d0ec700 1 --2- 172.21.15.88:0/1260861363 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6e980f5410 0x7f6e981056e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f6e880089f0 tx=0x7f6e8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:45.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:45.656+0000 7f6e8dffb700 1 -- 172.21.15.88:0/1260861363 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e88009320 con 0x7f6e980f5410 2024-02-19T22:12:45.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:45.656+0000 7f6e9f350700 1 -- 172.21.15.88:0/1260861363 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e98105dd0 con 0x7f6e980f5410 2024-02-19T22:12:45.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:45.656+0000 7f6e8dffb700 1 -- 172.21.15.88:0/1260861363 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6e88013380 con 0x7f6e980f5410 2024-02-19T22:12:45.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:45.656+0000 7f6e8dffb700 1 -- 172.21.15.88:0/1260861363 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e8801ed80 con 0x7f6e980f5410 2024-02-19T22:12:45.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:45.656+0000 7f6e9f350700 1 -- 172.21.15.88:0/1260861363 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e98106260 con 0x7f6e980f5410 2024-02-19T22:12:45.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:45.657+0000 7f6e8dffb700 1 -- 172.21.15.88:0/1260861363 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6e8801e420 con 0x7f6e980f5410 2024-02-19T22:12:45.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:45.657+0000 7f6e9f350700 1 -- 172.21.15.88:0/1260861363 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e98045ad0 con 0x7f6e980f5410 2024-02-19T22:12:45.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:45.657+0000 7f6e8dffb700 1 --2- 172.21.15.88:0/1260861363 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f6e8402ff60 0x7f6e84032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:45.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:45.657+0000 7f6e8dffb700 1 -- 172.21.15.88:0/1260861363 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6e88040ff0 con 0x7f6e980f5410 2024-02-19T22:12:45.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:45.657+0000 7f6e9c8eb700 1 --2- 172.21.15.88:0/1260861363 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f6e8402ff60 0x7f6e84032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:45.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:45.657+0000 7f6e9c8eb700 1 --2- 172.21.15.88:0/1260861363 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f6e8402ff60 0x7f6e84032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f6e940096a0 tx=0x7f6e94006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:45.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:45.662+0000 7f6e8dffb700 1 -- 172.21.15.88:0/1260861363 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6e880095f0 con 0x7f6e980f5410 2024-02-19T22:12:45.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:45.864+0000 7f6e9f350700 1 -- 172.21.15.88:0/1260861363 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6e98107110 con 0x7f6e980f5410 2024-02-19T22:12:45.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:45.864+0000 7f6e8dffb700 1 -- 172.21.15.88:0/1260861363 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6e8801cb10 con 0x7f6e980f5410 2024-02-19T22:12:45.866 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:12:45.866 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:12:45.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:45.866+0000 7f6e9f350700 1 -- 172.21.15.88:0/1260861363 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f6e8402ff60 msgr2=0x7f6e84032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:45.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:45.866+0000 7f6e9f350700 1 --2- 172.21.15.88:0/1260861363 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f6e8402ff60 0x7f6e84032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f6e940096a0 tx=0x7f6e94006b40).stop 2024-02-19T22:12:45.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:45.866+0000 7f6e9f350700 1 -- 172.21.15.88:0/1260861363 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6e980f5410 msgr2=0x7f6e981056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:45.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:45.866+0000 7f6e9f350700 1 --2- 172.21.15.88:0/1260861363 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6e980f5410 0x7f6e981056e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f6e880089f0 tx=0x7f6e8800b040).stop 2024-02-19T22:12:45.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:45.866+0000 7f6e9f350700 1 -- 172.21.15.88:0/1260861363 shutdown_connections 2024-02-19T22:12:45.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:45.866+0000 7f6e9f350700 1 --2- 172.21.15.88:0/1260861363 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f6e8402ff60 0x7f6e84032420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:45.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:45.866+0000 7f6e9f350700 1 --2- 172.21.15.88:0/1260861363 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6e980f5410 0x7f6e981056e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:45.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:45.866+0000 7f6e9f350700 1 -- 172.21.15.88:0/1260861363 >> 172.21.15.88:0/1260861363 conn(0x7f6e980f0b70 msgr2=0x7f6e980f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:45.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:45.866+0000 7f6e9f350700 1 -- 172.21.15.88:0/1260861363 shutdown_connections 2024-02-19T22:12:45.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:45.866+0000 7f6e9f350700 1 -- 172.21.15.88:0/1260861363 wait complete. 2024-02-19T22:12:45.869 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:12:46.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:12:45.865902+0000 mon.smithi052 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.88:0/1260861363' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:12:47.530 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:12:47.530 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:12:47.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:47 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:12:45.643386+0000 mgr.smithi052.ghqrtk (mgr.14162) 92 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:49.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:49.444+0000 7f2389565700 1 -- 172.21.15.88:0/3407090261 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f23840f6510 msgr2=0x7f23840f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:49.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:49.444+0000 7f2389565700 1 --2- 172.21.15.88:0/3407090261 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f23840f6510 0x7f23840f6930 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f236c009fb0 tx=0x7f236c009b30).stop 2024-02-19T22:12:49.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:49.444+0000 7f2389565700 1 -- 172.21.15.88:0/3407090261 shutdown_connections 2024-02-19T22:12:49.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:49.444+0000 7f2389565700 1 --2- 172.21.15.88:0/3407090261 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f23840f6510 0x7f23840f6930 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:49.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:49.444+0000 7f2389565700 1 -- 172.21.15.88:0/3407090261 >> 172.21.15.88:0/3407090261 conn(0x7f23840f1c70 msgr2=0x7f23840f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:49.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:49.445+0000 7f2389565700 1 -- 172.21.15.88:0/3407090261 shutdown_connections 2024-02-19T22:12:49.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:49.445+0000 7f2389565700 1 -- 172.21.15.88:0/3407090261 wait complete. 2024-02-19T22:12:49.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:49.445+0000 7f2389565700 1 Processor -- start 2024-02-19T22:12:49.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:49.445+0000 7f2389565700 1 -- start start 2024-02-19T22:12:49.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:49.445+0000 7f2389565700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f23840f6510 0x7f23841067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:49.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:49.445+0000 7f2389565700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2384106cd0 con 0x7f23840f6510 2024-02-19T22:12:49.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:49.446+0000 7f2382ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f23840f6510 0x7f23841067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:49.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:49.446+0000 7f2382ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f23840f6510 0x7f23841067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:46962/0 (socket says 172.21.15.88:46962) 2024-02-19T22:12:49.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:49.446+0000 7f2382ffd700 1 -- 172.21.15.88:0/308483632 learned_addr learned my addr 172.21.15.88:0/308483632 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:12:49.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:49.446+0000 7f2382ffd700 1 -- 172.21.15.88:0/308483632 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f236c005040 con 0x7f23840f6510 2024-02-19T22:12:49.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:49.446+0000 7f2382ffd700 1 --2- 172.21.15.88:0/308483632 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f23840f6510 0x7f23841067e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f236c005010 tx=0x7f236c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:49.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:49.446+0000 7f237b7fe700 1 -- 172.21.15.88:0/308483632 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f236c009320 con 0x7f23840f6510 2024-02-19T22:12:49.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:49.446+0000 7f237b7fe700 1 -- 172.21.15.88:0/308483632 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f236c013380 con 0x7f23840f6510 2024-02-19T22:12:49.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:49.446+0000 7f2389565700 1 -- 172.21.15.88:0/308483632 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2384106ed0 con 0x7f23840f6510 2024-02-19T22:12:49.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:49.446+0000 7f237b7fe700 1 -- 172.21.15.88:0/308483632 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f236c014e50 con 0x7f23840f6510 2024-02-19T22:12:49.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:49.446+0000 7f2389565700 1 -- 172.21.15.88:0/308483632 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2384107360 con 0x7f23840f6510 2024-02-19T22:12:49.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:49.447+0000 7f2389565700 1 -- 172.21.15.88:0/308483632 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2384045ad0 con 0x7f23840f6510 2024-02-19T22:12:49.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:49.447+0000 7f237b7fe700 1 -- 172.21.15.88:0/308483632 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f236c0095f0 con 0x7f23840f6510 2024-02-19T22:12:49.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:49.447+0000 7f237b7fe700 1 --2- 172.21.15.88:0/308483632 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f237002ff10 0x7f23700323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:49.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:49.447+0000 7f237b7fe700 1 -- 172.21.15.88:0/308483632 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f236c040f70 con 0x7f23840f6510 2024-02-19T22:12:49.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:49.448+0000 7f23827fc700 1 --2- 172.21.15.88:0/308483632 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f237002ff10 0x7f23700323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:49.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:49.448+0000 7f23827fc700 1 --2- 172.21.15.88:0/308483632 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f237002ff10 0x7f23700323d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f23740096a0 tx=0x7f2374006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:49.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:49.452+0000 7f237b7fe700 1 -- 172.21.15.88:0/308483632 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f236c014420 con 0x7f23840f6510 2024-02-19T22:12:49.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:49 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:12:47.643704+0000 mgr.smithi052.ghqrtk (mgr.14162) 93 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:49.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:49.656+0000 7f2389565700 1 -- 172.21.15.88:0/308483632 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2384108210 con 0x7f23840f6510 2024-02-19T22:12:49.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:49.657+0000 7f237b7fe700 1 -- 172.21.15.88:0/308483632 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f236c01d980 con 0x7f23840f6510 2024-02-19T22:12:49.659 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:12:49.659 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:12:49.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:49.658+0000 7f2389565700 1 -- 172.21.15.88:0/308483632 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f237002ff10 msgr2=0x7f23700323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:49.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:49.658+0000 7f2389565700 1 --2- 172.21.15.88:0/308483632 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f237002ff10 0x7f23700323d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f23740096a0 tx=0x7f2374006b40).stop 2024-02-19T22:12:49.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:49.658+0000 7f2389565700 1 -- 172.21.15.88:0/308483632 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f23840f6510 msgr2=0x7f23841067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:49.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:49.658+0000 7f2389565700 1 --2- 172.21.15.88:0/308483632 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f23840f6510 0x7f23841067e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f236c005010 tx=0x7f236c00b040).stop 2024-02-19T22:12:49.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:49.659+0000 7f2389565700 1 -- 172.21.15.88:0/308483632 shutdown_connections 2024-02-19T22:12:49.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:49.659+0000 7f2389565700 1 --2- 172.21.15.88:0/308483632 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f237002ff10 0x7f23700323d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:49.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:49.659+0000 7f2389565700 1 --2- 172.21.15.88:0/308483632 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f23840f6510 0x7f23841067e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:49.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:49.659+0000 7f2389565700 1 -- 172.21.15.88:0/308483632 >> 172.21.15.88:0/308483632 conn(0x7f23840f1c70 msgr2=0x7f23840f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:49.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:49.659+0000 7f2389565700 1 -- 172.21.15.88:0/308483632 shutdown_connections 2024-02-19T22:12:49.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:49.659+0000 7f2389565700 1 -- 172.21.15.88:0/308483632 wait complete. 2024-02-19T22:12:49.662 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:12:50.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:50 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:12:49.658448+0000 mon.smithi052 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.88:0/308483632' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:12:51.302 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:12:51.302 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:12:51.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:51 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:12:49.644047+0000 mgr.smithi052.ghqrtk (mgr.14162) 94 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:53.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:53.127+0000 7f06f68f3700 1 -- 172.21.15.88:0/1372581462 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f06f00f5410 msgr2=0x7f06f00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:53.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:53.127+0000 7f06f68f3700 1 --2- 172.21.15.88:0/1372581462 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f06f00f5410 0x7f06f00f5830 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f06e4009fb0 tx=0x7f06e4009b30).stop 2024-02-19T22:12:53.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:53.127+0000 7f06f68f3700 1 -- 172.21.15.88:0/1372581462 shutdown_connections 2024-02-19T22:12:53.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:53.127+0000 7f06f68f3700 1 --2- 172.21.15.88:0/1372581462 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f06f00f5410 0x7f06f00f5830 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:53.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:53.127+0000 7f06f68f3700 1 -- 172.21.15.88:0/1372581462 >> 172.21.15.88:0/1372581462 conn(0x7f06f00f0b70 msgr2=0x7f06f00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:53.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:53.127+0000 7f06f68f3700 1 -- 172.21.15.88:0/1372581462 shutdown_connections 2024-02-19T22:12:53.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:53.127+0000 7f06f68f3700 1 -- 172.21.15.88:0/1372581462 wait complete. 2024-02-19T22:12:53.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:53.127+0000 7f06f68f3700 1 Processor -- start 2024-02-19T22:12:53.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:53.127+0000 7f06f68f3700 1 -- start start 2024-02-19T22:12:53.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:53.128+0000 7f06f68f3700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f06f00f5410 0x7f06f01056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:53.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:53.128+0000 7f06f68f3700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f0105bd0 con 0x7f06f00f5410 2024-02-19T22:12:53.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:53.128+0000 7f06effff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f06f00f5410 0x7f06f01056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:53.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:53.128+0000 7f06effff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f06f00f5410 0x7f06f01056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:46980/0 (socket says 172.21.15.88:46980) 2024-02-19T22:12:53.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:53.128+0000 7f06effff700 1 -- 172.21.15.88:0/3364298977 learned_addr learned my addr 172.21.15.88:0/3364298977 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:12:53.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:53.128+0000 7f06effff700 1 -- 172.21.15.88:0/3364298977 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06e4005040 con 0x7f06f00f5410 2024-02-19T22:12:53.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:53.128+0000 7f06effff700 1 --2- 172.21.15.88:0/3364298977 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f06f00f5410 0x7f06f01056e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f06e4005010 tx=0x7f06e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:53.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:53.128+0000 7f06ecff9700 1 -- 172.21.15.88:0/3364298977 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06e4009320 con 0x7f06f00f5410 2024-02-19T22:12:53.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:53.128+0000 7f06f68f3700 1 -- 172.21.15.88:0/3364298977 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06f0105dd0 con 0x7f06f00f5410 2024-02-19T22:12:53.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:53.128+0000 7f06ecff9700 1 -- 172.21.15.88:0/3364298977 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f06e4013380 con 0x7f06f00f5410 2024-02-19T22:12:53.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:53.128+0000 7f06ecff9700 1 -- 172.21.15.88:0/3364298977 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06e4014e50 con 0x7f06f00f5410 2024-02-19T22:12:53.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:53.129+0000 7f06f68f3700 1 -- 172.21.15.88:0/3364298977 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06f0106260 con 0x7f06f00f5410 2024-02-19T22:12:53.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:53.129+0000 7f06ecff9700 1 -- 172.21.15.88:0/3364298977 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f06e40095f0 con 0x7f06f00f5410 2024-02-19T22:12:53.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:53.129+0000 7f06f68f3700 1 -- 172.21.15.88:0/3364298977 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06f0045ad0 con 0x7f06f00f5410 2024-02-19T22:12:53.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:53.130+0000 7f06ecff9700 1 --2- 172.21.15.88:0/3364298977 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f06d802ff10 0x7f06d80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:53.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:53.130+0000 7f06ecff9700 1 -- 172.21.15.88:0/3364298977 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f06e4040e90 con 0x7f06f00f5410 2024-02-19T22:12:53.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:53.130+0000 7f06ef7fe700 1 --2- 172.21.15.88:0/3364298977 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f06d802ff10 0x7f06d80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:53.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:53.130+0000 7f06ef7fe700 1 --2- 172.21.15.88:0/3364298977 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f06d802ff10 0x7f06d80323d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f06e00096a0 tx=0x7f06e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:53.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:53.134+0000 7f06ecff9700 1 -- 172.21.15.88:0/3364298977 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f06e4014420 con 0x7f06f00f5410 2024-02-19T22:12:53.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:53.338+0000 7f06f68f3700 1 -- 172.21.15.88:0/3364298977 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f06f0107110 con 0x7f06f00f5410 2024-02-19T22:12:53.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:53.339+0000 7f06ecff9700 1 -- 172.21.15.88:0/3364298977 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f06e401d980 con 0x7f06f00f5410 2024-02-19T22:12:53.340 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:12:53.341 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:12:53.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:53.340+0000 7f06f68f3700 1 -- 172.21.15.88:0/3364298977 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f06d802ff10 msgr2=0x7f06d80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:53.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:53.340+0000 7f06f68f3700 1 --2- 172.21.15.88:0/3364298977 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f06d802ff10 0x7f06d80323d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f06e00096a0 tx=0x7f06e0006b40).stop 2024-02-19T22:12:53.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:53.340+0000 7f06f68f3700 1 -- 172.21.15.88:0/3364298977 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f06f00f5410 msgr2=0x7f06f01056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:53.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:53.340+0000 7f06f68f3700 1 --2- 172.21.15.88:0/3364298977 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f06f00f5410 0x7f06f01056e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f06e4005010 tx=0x7f06e400b040).stop 2024-02-19T22:12:53.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:53.340+0000 7f06f68f3700 1 -- 172.21.15.88:0/3364298977 shutdown_connections 2024-02-19T22:12:53.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:53.340+0000 7f06f68f3700 1 --2- 172.21.15.88:0/3364298977 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f06d802ff10 0x7f06d80323d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:53.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:53.340+0000 7f06f68f3700 1 --2- 172.21.15.88:0/3364298977 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f06f00f5410 0x7f06f01056e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:53.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:53.340+0000 7f06f68f3700 1 -- 172.21.15.88:0/3364298977 >> 172.21.15.88:0/3364298977 conn(0x7f06f00f0b70 msgr2=0x7f06f00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:53.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:53.340+0000 7f06f68f3700 1 -- 172.21.15.88:0/3364298977 shutdown_connections 2024-02-19T22:12:53.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:53.340+0000 7f06f68f3700 1 -- 172.21.15.88:0/3364298977 wait complete. 2024-02-19T22:12:53.343 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:12:53.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:53 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:12:51.644470+0000 mgr.smithi052.ghqrtk (mgr.14162) 95 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:54.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:54 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:12:53.340099+0000 mon.smithi052 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.88:0/3364298977' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:12:54.999 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:12:54.999 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:12:55.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:55 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:12:53.644878+0000 mgr.smithi052.ghqrtk (mgr.14162) 96 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:56.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:56.889+0000 7fbeaaba3700 1 -- 172.21.15.88:0/1355542744 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbea40f51a0 msgr2=0x7fbea40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:56.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:56.889+0000 7fbeaaba3700 1 --2- 172.21.15.88:0/1355542744 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbea40f51a0 0x7fbea40f55c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fbe94009fb0 tx=0x7fbe94009b30).stop 2024-02-19T22:12:56.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:56.890+0000 7fbeaaba3700 1 -- 172.21.15.88:0/1355542744 shutdown_connections 2024-02-19T22:12:56.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:56.890+0000 7fbeaaba3700 1 --2- 172.21.15.88:0/1355542744 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbea40f51a0 0x7fbea40f55c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:56.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:56.890+0000 7fbeaaba3700 1 -- 172.21.15.88:0/1355542744 >> 172.21.15.88:0/1355542744 conn(0x7fbea40f09a0 msgr2=0x7fbea40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:56.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:56.890+0000 7fbeaaba3700 1 -- 172.21.15.88:0/1355542744 shutdown_connections 2024-02-19T22:12:56.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:56.890+0000 7fbeaaba3700 1 -- 172.21.15.88:0/1355542744 wait complete. 2024-02-19T22:12:56.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:56.890+0000 7fbeaaba3700 1 Processor -- start 2024-02-19T22:12:56.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:56.890+0000 7fbeaaba3700 1 -- start start 2024-02-19T22:12:56.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:56.890+0000 7fbeaaba3700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbea40f51a0 0x7fbea41065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:56.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:56.890+0000 7fbeaaba3700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbea4106ac0 con 0x7fbea40f51a0 2024-02-19T22:12:56.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:56.890+0000 7fbea9ba1700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbea40f51a0 0x7fbea41065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:56.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:56.890+0000 7fbea9ba1700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbea40f51a0 0x7fbea41065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:45296/0 (socket says 172.21.15.88:45296) 2024-02-19T22:12:56.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:56.890+0000 7fbea9ba1700 1 -- 172.21.15.88:0/257883126 learned_addr learned my addr 172.21.15.88:0/257883126 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:12:56.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:56.891+0000 7fbea9ba1700 1 -- 172.21.15.88:0/257883126 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe94005040 con 0x7fbea40f51a0 2024-02-19T22:12:56.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:56.891+0000 7fbea9ba1700 1 --2- 172.21.15.88:0/257883126 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbea40f51a0 0x7fbea41065d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fbe940089f0 tx=0x7fbe9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:56.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:56.891+0000 7fbe9a7fc700 1 -- 172.21.15.88:0/257883126 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe94009320 con 0x7fbea40f51a0 2024-02-19T22:12:56.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:56.891+0000 7fbe9a7fc700 1 -- 172.21.15.88:0/257883126 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbe94013380 con 0x7fbea40f51a0 2024-02-19T22:12:56.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:56.891+0000 7fbeaaba3700 1 -- 172.21.15.88:0/257883126 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbea4106cc0 con 0x7fbea40f51a0 2024-02-19T22:12:56.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:56.891+0000 7fbe9a7fc700 1 -- 172.21.15.88:0/257883126 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe94014e20 con 0x7fbea40f51a0 2024-02-19T22:12:56.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:56.891+0000 7fbeaaba3700 1 -- 172.21.15.88:0/257883126 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbea4107150 con 0x7fbea40f51a0 2024-02-19T22:12:56.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:56.892+0000 7fbe9a7fc700 1 -- 172.21.15.88:0/257883126 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbe940095f0 con 0x7fbea40f51a0 2024-02-19T22:12:56.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:56.892+0000 7fbeaaba3700 1 -- 172.21.15.88:0/257883126 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbea4045ad0 con 0x7fbea40f51a0 2024-02-19T22:12:56.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:56.892+0000 7fbe9a7fc700 1 --2- 172.21.15.88:0/257883126 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fbe9002ff10 0x7fbe900323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:56.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:56.892+0000 7fbe9a7fc700 1 -- 172.21.15.88:0/257883126 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbe94040f50 con 0x7fbea40f51a0 2024-02-19T22:12:56.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:56.892+0000 7fbea93a0700 1 --2- 172.21.15.88:0/257883126 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fbe9002ff10 0x7fbe900323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:56.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:56.893+0000 7fbea93a0700 1 --2- 172.21.15.88:0/257883126 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fbe9002ff10 0x7fbe900323d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fbea00096a0 tx=0x7fbea0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:56.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:56.897+0000 7fbe9a7fc700 1 -- 172.21.15.88:0/257883126 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbe940268c0 con 0x7fbea40f51a0 2024-02-19T22:12:57.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:57.101+0000 7fbeaaba3700 1 -- 172.21.15.88:0/257883126 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbea40fad30 con 0x7fbea40f51a0 2024-02-19T22:12:57.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:57.101+0000 7fbe9a7fc700 1 -- 172.21.15.88:0/257883126 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbe9401d980 con 0x7fbea40f51a0 2024-02-19T22:12:57.103 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:12:57.104 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:12:57.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:57.103+0000 7fbeaaba3700 1 -- 172.21.15.88:0/257883126 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fbe9002ff10 msgr2=0x7fbe900323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:57.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:57.103+0000 7fbeaaba3700 1 --2- 172.21.15.88:0/257883126 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fbe9002ff10 0x7fbe900323d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fbea00096a0 tx=0x7fbea0006b40).stop 2024-02-19T22:12:57.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:57.103+0000 7fbeaaba3700 1 -- 172.21.15.88:0/257883126 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbea40f51a0 msgr2=0x7fbea41065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:57.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:57.103+0000 7fbeaaba3700 1 --2- 172.21.15.88:0/257883126 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbea40f51a0 0x7fbea41065d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fbe940089f0 tx=0x7fbe9400b040).stop 2024-02-19T22:12:57.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:57.103+0000 7fbeaaba3700 1 -- 172.21.15.88:0/257883126 shutdown_connections 2024-02-19T22:12:57.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:57.103+0000 7fbeaaba3700 1 --2- 172.21.15.88:0/257883126 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fbe9002ff10 0x7fbe900323d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:57.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:57.103+0000 7fbeaaba3700 1 --2- 172.21.15.88:0/257883126 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbea40f51a0 0x7fbea41065d0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:57.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:57.103+0000 7fbeaaba3700 1 -- 172.21.15.88:0/257883126 >> 172.21.15.88:0/257883126 conn(0x7fbea40f09a0 msgr2=0x7fbea405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:57.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:57.103+0000 7fbeaaba3700 1 -- 172.21.15.88:0/257883126 shutdown_connections 2024-02-19T22:12:57.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:12:57.103+0000 7fbeaaba3700 1 -- 172.21.15.88:0/257883126 wait complete. 2024-02-19T22:12:57.106 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:12:57.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:57 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:12:55.645263+0000 mgr.smithi052.ghqrtk (mgr.14162) 97 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:57.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:57 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:12:57.102710+0000 mon.smithi052 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.88:0/257883126' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:12:58.728 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:12:58.728 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:12:59.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:12:59 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:12:57.645615+0000 mgr.smithi052.ghqrtk (mgr.14162) 98 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:00.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:00.563+0000 7fc1ac785700 1 -- 172.21.15.88:0/1277546779 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc1a40f5410 msgr2=0x7fc1a40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:00.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:00.563+0000 7fc1ac785700 1 --2- 172.21.15.88:0/1277546779 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc1a40f5410 0x7fc1a40f5830 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fc194009fb0 tx=0x7fc194009b30).stop 2024-02-19T22:13:00.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:00.563+0000 7fc1ac785700 1 -- 172.21.15.88:0/1277546779 shutdown_connections 2024-02-19T22:13:00.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:00.563+0000 7fc1ac785700 1 --2- 172.21.15.88:0/1277546779 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc1a40f5410 0x7fc1a40f5830 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:00.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:00.563+0000 7fc1ac785700 1 -- 172.21.15.88:0/1277546779 >> 172.21.15.88:0/1277546779 conn(0x7fc1a40f0b70 msgr2=0x7fc1a40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:00.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:00.563+0000 7fc1ac785700 1 -- 172.21.15.88:0/1277546779 shutdown_connections 2024-02-19T22:13:00.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:00.563+0000 7fc1ac785700 1 -- 172.21.15.88:0/1277546779 wait complete. 2024-02-19T22:13:00.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:00.564+0000 7fc1ac785700 1 Processor -- start 2024-02-19T22:13:00.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:00.564+0000 7fc1ac785700 1 -- start start 2024-02-19T22:13:00.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:00.564+0000 7fc1ac785700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc1a40f5410 0x7fc1a41056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:00.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:00.564+0000 7fc1ac785700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a4105bd0 con 0x7fc1a40f5410 2024-02-19T22:13:00.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:00.564+0000 7fc1aa521700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc1a40f5410 0x7fc1a41056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:00.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:00.564+0000 7fc1aa521700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc1a40f5410 0x7fc1a41056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:45304/0 (socket says 172.21.15.88:45304) 2024-02-19T22:13:00.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:00.564+0000 7fc1aa521700 1 -- 172.21.15.88:0/68483212 learned_addr learned my addr 172.21.15.88:0/68483212 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:13:00.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:00.564+0000 7fc1aa521700 1 -- 172.21.15.88:0/68483212 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc194005040 con 0x7fc1a40f5410 2024-02-19T22:13:00.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:00.565+0000 7fc1aa521700 1 --2- 172.21.15.88:0/68483212 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc1a40f5410 0x7fc1a41056e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fc1940089f0 tx=0x7fc19400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:00.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:00.565+0000 7fc19affd700 1 -- 172.21.15.88:0/68483212 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc194009320 con 0x7fc1a40f5410 2024-02-19T22:13:00.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:00.565+0000 7fc1ac785700 1 -- 172.21.15.88:0/68483212 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1a4105dd0 con 0x7fc1a40f5410 2024-02-19T22:13:00.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:00.565+0000 7fc19affd700 1 -- 172.21.15.88:0/68483212 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc194013380 con 0x7fc1a40f5410 2024-02-19T22:13:00.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:00.565+0000 7fc19affd700 1 -- 172.21.15.88:0/68483212 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc19401ed80 con 0x7fc1a40f5410 2024-02-19T22:13:00.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:00.565+0000 7fc1ac785700 1 -- 172.21.15.88:0/68483212 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1a4106260 con 0x7fc1a40f5410 2024-02-19T22:13:00.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:00.566+0000 7fc19affd700 1 -- 172.21.15.88:0/68483212 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc19401e420 con 0x7fc1a40f5410 2024-02-19T22:13:00.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:00.566+0000 7fc1ac785700 1 -- 172.21.15.88:0/68483212 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1a4045ad0 con 0x7fc1a40f5410 2024-02-19T22:13:00.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:00.566+0000 7fc19affd700 1 --2- 172.21.15.88:0/68483212 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fc19002ff60 0x7fc190032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:00.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:00.566+0000 7fc19affd700 1 -- 172.21.15.88:0/68483212 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc194040e70 con 0x7fc1a40f5410 2024-02-19T22:13:00.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:00.566+0000 7fc1a9d20700 1 --2- 172.21.15.88:0/68483212 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fc19002ff60 0x7fc190032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:00.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:00.566+0000 7fc1a9d20700 1 --2- 172.21.15.88:0/68483212 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fc19002ff60 0x7fc190032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fc1a00096a0 tx=0x7fc1a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:00.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:00.571+0000 7fc19affd700 1 -- 172.21.15.88:0/68483212 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc19400e2a0 con 0x7fc1a40f5410 2024-02-19T22:13:00.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:00.776+0000 7fc1ac785700 1 -- 172.21.15.88:0/68483212 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc1a4107110 con 0x7fc1a40f5410 2024-02-19T22:13:00.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:00.776+0000 7fc19affd700 1 -- 172.21.15.88:0/68483212 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc19400e2a0 con 0x7fc1a40f5410 2024-02-19T22:13:00.778 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:13:00.778 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:13:00.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:00.778+0000 7fc1ac785700 1 -- 172.21.15.88:0/68483212 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fc19002ff60 msgr2=0x7fc190032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:00.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:00.778+0000 7fc1ac785700 1 --2- 172.21.15.88:0/68483212 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fc19002ff60 0x7fc190032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fc1a00096a0 tx=0x7fc1a0006b40).stop 2024-02-19T22:13:00.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:00.778+0000 7fc1ac785700 1 -- 172.21.15.88:0/68483212 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc1a40f5410 msgr2=0x7fc1a41056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:00.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:00.778+0000 7fc1ac785700 1 --2- 172.21.15.88:0/68483212 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc1a40f5410 0x7fc1a41056e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fc1940089f0 tx=0x7fc19400b040).stop 2024-02-19T22:13:00.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:00.778+0000 7fc1ac785700 1 -- 172.21.15.88:0/68483212 shutdown_connections 2024-02-19T22:13:00.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:00.778+0000 7fc1ac785700 1 --2- 172.21.15.88:0/68483212 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fc19002ff60 0x7fc190032420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:00.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:00.778+0000 7fc1ac785700 1 --2- 172.21.15.88:0/68483212 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc1a40f5410 0x7fc1a41056e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:00.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:00.778+0000 7fc1ac785700 1 -- 172.21.15.88:0/68483212 >> 172.21.15.88:0/68483212 conn(0x7fc1a40f0b70 msgr2=0x7fc1a40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:00.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:00.778+0000 7fc1ac785700 1 -- 172.21.15.88:0/68483212 shutdown_connections 2024-02-19T22:13:00.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:00.778+0000 7fc1ac785700 1 -- 172.21.15.88:0/68483212 wait complete. 2024-02-19T22:13:00.781 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:13:01.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:01 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:12:59.645933+0000 mgr.smithi052.ghqrtk (mgr.14162) 99 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:01.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:01 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:13:00.777955+0000 mon.smithi052 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.88:0/68483212' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:13:02.412 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:13:02.412 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:13:03.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:03 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:13:01.646283+0000 mgr.smithi052.ghqrtk (mgr.14162) 100 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:04.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:04.239+0000 7fd889da0700 1 -- 172.21.15.88:0/1617291397 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8840f51a0 msgr2=0x7fd8840f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:04.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:04.239+0000 7fd889da0700 1 --2- 172.21.15.88:0/1617291397 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8840f51a0 0x7fd8840f55c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fd878009fb0 tx=0x7fd878009b30).stop 2024-02-19T22:13:04.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:04.239+0000 7fd889da0700 1 -- 172.21.15.88:0/1617291397 shutdown_connections 2024-02-19T22:13:04.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:04.239+0000 7fd889da0700 1 --2- 172.21.15.88:0/1617291397 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8840f51a0 0x7fd8840f55c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:04.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:04.239+0000 7fd889da0700 1 -- 172.21.15.88:0/1617291397 >> 172.21.15.88:0/1617291397 conn(0x7fd8840f09a0 msgr2=0x7fd8840f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:04.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:04.239+0000 7fd889da0700 1 -- 172.21.15.88:0/1617291397 shutdown_connections 2024-02-19T22:13:04.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:04.239+0000 7fd889da0700 1 -- 172.21.15.88:0/1617291397 wait complete. 2024-02-19T22:13:04.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:04.239+0000 7fd889da0700 1 Processor -- start 2024-02-19T22:13:04.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:04.239+0000 7fd889da0700 1 -- start start 2024-02-19T22:13:04.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:04.240+0000 7fd889da0700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8840f51a0 0x7fd8841054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:04.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:04.240+0000 7fd889da0700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8841059c0 con 0x7fd8840f51a0 2024-02-19T22:13:04.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:04.240+0000 7fd888d9e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8840f51a0 0x7fd8841054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:04.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:04.240+0000 7fd888d9e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8840f51a0 0x7fd8841054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:34816/0 (socket says 172.21.15.88:34816) 2024-02-19T22:13:04.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:04.240+0000 7fd888d9e700 1 -- 172.21.15.88:0/3440762202 learned_addr learned my addr 172.21.15.88:0/3440762202 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:13:04.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:04.241+0000 7fd888d9e700 1 -- 172.21.15.88:0/3440762202 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd878005040 con 0x7fd8840f51a0 2024-02-19T22:13:04.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:04.241+0000 7fd888d9e700 1 --2- 172.21.15.88:0/3440762202 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8840f51a0 0x7fd8841054d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fd8780089f0 tx=0x7fd87800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:04.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:04.241+0000 7fd8817fa700 1 -- 172.21.15.88:0/3440762202 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd878009320 con 0x7fd8840f51a0 2024-02-19T22:13:04.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:04.241+0000 7fd8817fa700 1 -- 172.21.15.88:0/3440762202 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd878013380 con 0x7fd8840f51a0 2024-02-19T22:13:04.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:04.241+0000 7fd889da0700 1 -- 172.21.15.88:0/3440762202 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd884105bc0 con 0x7fd8840f51a0 2024-02-19T22:13:04.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:04.241+0000 7fd8817fa700 1 -- 172.21.15.88:0/3440762202 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd878014e20 con 0x7fd8840f51a0 2024-02-19T22:13:04.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:04.241+0000 7fd889da0700 1 -- 172.21.15.88:0/3440762202 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd884106050 con 0x7fd8840f51a0 2024-02-19T22:13:04.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:04.242+0000 7fd8817fa700 1 -- 172.21.15.88:0/3440762202 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd8780095f0 con 0x7fd8840f51a0 2024-02-19T22:13:04.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:04.242+0000 7fd889da0700 1 -- 172.21.15.88:0/3440762202 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd884045ad0 con 0x7fd8840f51a0 2024-02-19T22:13:04.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:04.242+0000 7fd8817fa700 1 --2- 172.21.15.88:0/3440762202 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd87402ff10 0x7fd8740323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:04.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:04.242+0000 7fd8817fa700 1 -- 172.21.15.88:0/3440762202 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd878040f50 con 0x7fd8840f51a0 2024-02-19T22:13:04.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:04.242+0000 7fd883fff700 1 --2- 172.21.15.88:0/3440762202 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd87402ff10 0x7fd8740323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:04.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:04.242+0000 7fd883fff700 1 --2- 172.21.15.88:0/3440762202 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd87402ff10 0x7fd8740323d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fd8700096a0 tx=0x7fd870006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:04.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:04.247+0000 7fd8817fa700 1 -- 172.21.15.88:0/3440762202 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd878014420 con 0x7fd8840f51a0 2024-02-19T22:13:04.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:04.451+0000 7fd889da0700 1 -- 172.21.15.88:0/3440762202 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd884106f00 con 0x7fd8840f51a0 2024-02-19T22:13:04.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:04.451+0000 7fd8817fa700 1 -- 172.21.15.88:0/3440762202 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd87801d980 con 0x7fd8840f51a0 2024-02-19T22:13:04.453 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:13:04.453 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:13:04.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:04.453+0000 7fd889da0700 1 -- 172.21.15.88:0/3440762202 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd87402ff10 msgr2=0x7fd8740323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:04.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:04.453+0000 7fd889da0700 1 --2- 172.21.15.88:0/3440762202 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd87402ff10 0x7fd8740323d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fd8700096a0 tx=0x7fd870006b40).stop 2024-02-19T22:13:04.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:04.453+0000 7fd889da0700 1 -- 172.21.15.88:0/3440762202 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8840f51a0 msgr2=0x7fd8841054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:04.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:04.453+0000 7fd889da0700 1 --2- 172.21.15.88:0/3440762202 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8840f51a0 0x7fd8841054d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fd8780089f0 tx=0x7fd87800b040).stop 2024-02-19T22:13:04.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:04.453+0000 7fd889da0700 1 -- 172.21.15.88:0/3440762202 shutdown_connections 2024-02-19T22:13:04.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:04.453+0000 7fd889da0700 1 --2- 172.21.15.88:0/3440762202 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd87402ff10 0x7fd8740323d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:04.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:04.453+0000 7fd889da0700 1 --2- 172.21.15.88:0/3440762202 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8840f51a0 0x7fd8841054d0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:04.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:04.453+0000 7fd889da0700 1 -- 172.21.15.88:0/3440762202 >> 172.21.15.88:0/3440762202 conn(0x7fd8840f09a0 msgr2=0x7fd8840f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:04.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:04.453+0000 7fd889da0700 1 -- 172.21.15.88:0/3440762202 shutdown_connections 2024-02-19T22:13:04.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:04.453+0000 7fd889da0700 1 -- 172.21.15.88:0/3440762202 wait complete. 2024-02-19T22:13:04.456 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:13:05.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:05 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:13:03.646606+0000 mgr.smithi052.ghqrtk (mgr.14162) 101 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:05.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:05 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:13:04.452884+0000 mon.smithi052 (mon.0) 167 : audit [DBG] from='client.? 172.21.15.88:0/3440762202' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:13:06.081 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:13:06.082 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:13:07.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:07 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:13:05.646905+0000 mgr.smithi052.ghqrtk (mgr.14162) 102 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:07.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:07.885+0000 7f70c9487700 1 -- 172.21.15.88:0/3631332877 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f70c40f5410 msgr2=0x7f70c40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:07.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:07.885+0000 7f70c9487700 1 --2- 172.21.15.88:0/3631332877 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f70c40f5410 0x7f70c40f5830 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f70b4009fb0 tx=0x7f70b4009b30).stop 2024-02-19T22:13:07.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:07.885+0000 7f70c9487700 1 -- 172.21.15.88:0/3631332877 shutdown_connections 2024-02-19T22:13:07.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:07.885+0000 7f70c9487700 1 --2- 172.21.15.88:0/3631332877 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f70c40f5410 0x7f70c40f5830 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:07.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:07.885+0000 7f70c9487700 1 -- 172.21.15.88:0/3631332877 >> 172.21.15.88:0/3631332877 conn(0x7f70c40f0b70 msgr2=0x7f70c40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:07.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:07.886+0000 7f70c9487700 1 -- 172.21.15.88:0/3631332877 shutdown_connections 2024-02-19T22:13:07.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:07.886+0000 7f70c9487700 1 -- 172.21.15.88:0/3631332877 wait complete. 2024-02-19T22:13:07.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:07.886+0000 7f70c9487700 1 Processor -- start 2024-02-19T22:13:07.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:07.886+0000 7f70c9487700 1 -- start start 2024-02-19T22:13:07.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:07.887+0000 7f70c9487700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f70c40f5410 0x7f70c4109520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:07.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:07.887+0000 7f70c9487700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c4060240 con 0x7f70c40f5410 2024-02-19T22:13:07.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:07.887+0000 7f70c2ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f70c40f5410 0x7f70c4109520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:07.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:07.887+0000 7f70c2ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f70c40f5410 0x7f70c4109520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:34832/0 (socket says 172.21.15.88:34832) 2024-02-19T22:13:07.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:07.887+0000 7f70c2ffd700 1 -- 172.21.15.88:0/453856552 learned_addr learned my addr 172.21.15.88:0/453856552 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:13:07.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:07.887+0000 7f70c2ffd700 1 -- 172.21.15.88:0/453856552 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70b4005040 con 0x7f70c40f5410 2024-02-19T22:13:07.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:07.887+0000 7f70c2ffd700 1 --2- 172.21.15.88:0/453856552 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f70c40f5410 0x7f70c4109520 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f70b4005010 tx=0x7f70b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:07.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:07.887+0000 7f70ab7fe700 1 -- 172.21.15.88:0/453856552 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70b4009320 con 0x7f70c40f5410 2024-02-19T22:13:07.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:07.887+0000 7f70c9487700 1 -- 172.21.15.88:0/453856552 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70c4060440 con 0x7f70c40f5410 2024-02-19T22:13:07.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:07.887+0000 7f70ab7fe700 1 -- 172.21.15.88:0/453856552 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f70b4013380 con 0x7f70c40f5410 2024-02-19T22:13:07.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:07.888+0000 7f70ab7fe700 1 -- 172.21.15.88:0/453856552 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70b4014e50 con 0x7f70c40f5410 2024-02-19T22:13:07.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:07.888+0000 7f70c9487700 1 -- 172.21.15.88:0/453856552 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70c4109bd0 con 0x7f70c40f5410 2024-02-19T22:13:07.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:07.888+0000 7f70ab7fe700 1 -- 172.21.15.88:0/453856552 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f70b40095f0 con 0x7f70c40f5410 2024-02-19T22:13:07.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:07.888+0000 7f70c9487700 1 -- 172.21.15.88:0/453856552 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70c4045ad0 con 0x7f70c40f5410 2024-02-19T22:13:07.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:07.888+0000 7f70ab7fe700 1 --2- 172.21.15.88:0/453856552 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f70ac02ff10 0x7f70ac0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:07.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:07.888+0000 7f70ab7fe700 1 -- 172.21.15.88:0/453856552 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f70b4040e90 con 0x7f70c40f5410 2024-02-19T22:13:07.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:07.889+0000 7f70c27fc700 1 --2- 172.21.15.88:0/453856552 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f70ac02ff10 0x7f70ac0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:07.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:07.889+0000 7f70c27fc700 1 --2- 172.21.15.88:0/453856552 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f70ac02ff10 0x7f70ac0323d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f70b80096a0 tx=0x7f70b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:07.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:07.893+0000 7f70ab7fe700 1 -- 172.21.15.88:0/453856552 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f70b4014420 con 0x7f70c40f5410 2024-02-19T22:13:08.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:08.094+0000 7f70c9487700 1 -- 172.21.15.88:0/453856552 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f70c410a3f0 con 0x7f70c40f5410 2024-02-19T22:13:08.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:08.094+0000 7f70ab7fe700 1 -- 172.21.15.88:0/453856552 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f70b4010230 con 0x7f70c40f5410 2024-02-19T22:13:08.096 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:13:08.096 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:13:08.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:08.096+0000 7f70c9487700 1 -- 172.21.15.88:0/453856552 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f70ac02ff10 msgr2=0x7f70ac0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:08.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:08.096+0000 7f70c9487700 1 --2- 172.21.15.88:0/453856552 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f70ac02ff10 0x7f70ac0323d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f70b80096a0 tx=0x7f70b8006b40).stop 2024-02-19T22:13:08.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:08.096+0000 7f70c9487700 1 -- 172.21.15.88:0/453856552 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f70c40f5410 msgr2=0x7f70c4109520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:08.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:08.096+0000 7f70c9487700 1 --2- 172.21.15.88:0/453856552 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f70c40f5410 0x7f70c4109520 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f70b4005010 tx=0x7f70b400b040).stop 2024-02-19T22:13:08.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:08.096+0000 7f70c9487700 1 -- 172.21.15.88:0/453856552 shutdown_connections 2024-02-19T22:13:08.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:08.096+0000 7f70c9487700 1 --2- 172.21.15.88:0/453856552 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f70ac02ff10 0x7f70ac0323d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:08.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:08.096+0000 7f70c9487700 1 --2- 172.21.15.88:0/453856552 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f70c40f5410 0x7f70c4109520 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:08.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:08.096+0000 7f70c9487700 1 -- 172.21.15.88:0/453856552 >> 172.21.15.88:0/453856552 conn(0x7f70c40f0b70 msgr2=0x7f70c40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:08.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:08.096+0000 7f70c9487700 1 -- 172.21.15.88:0/453856552 shutdown_connections 2024-02-19T22:13:08.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:08.096+0000 7f70c9487700 1 -- 172.21.15.88:0/453856552 wait complete. 2024-02-19T22:13:08.098 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:13:08.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:08 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:13:08.094991+0000 mon.smithi052 (mon.0) 168 : audit [DBG] from='client.? 172.21.15.88:0/453856552' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:13:09.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:09 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:13:07.647298+0000 mgr.smithi052.ghqrtk (mgr.14162) 103 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:09.755 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:13:09.755 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:13:11.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:11.551+0000 7f28f0921700 1 -- 172.21.15.88:0/1920546666 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f28ec0f6510 msgr2=0x7f28ec0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:11.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:11.551+0000 7f28f0921700 1 --2- 172.21.15.88:0/1920546666 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f28ec0f6510 0x7f28ec0f6930 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f28dc009fb0 tx=0x7f28dc009b30).stop 2024-02-19T22:13:11.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:11.551+0000 7f28f0921700 1 -- 172.21.15.88:0/1920546666 shutdown_connections 2024-02-19T22:13:11.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:11.551+0000 7f28f0921700 1 --2- 172.21.15.88:0/1920546666 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f28ec0f6510 0x7f28ec0f6930 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:11.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:11.551+0000 7f28f0921700 1 -- 172.21.15.88:0/1920546666 >> 172.21.15.88:0/1920546666 conn(0x7f28ec0f1c70 msgr2=0x7f28ec0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:11.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:11.551+0000 7f28f0921700 1 -- 172.21.15.88:0/1920546666 shutdown_connections 2024-02-19T22:13:11.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:11.551+0000 7f28f0921700 1 -- 172.21.15.88:0/1920546666 wait complete. 2024-02-19T22:13:11.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:11.552+0000 7f28f0921700 1 Processor -- start 2024-02-19T22:13:11.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:11.552+0000 7f28f0921700 1 -- start start 2024-02-19T22:13:11.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:11.552+0000 7f28f0921700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f28ec0f6510 0x7f28ec1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:11.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:11.552+0000 7f28f0921700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28ec106cd0 con 0x7f28ec0f6510 2024-02-19T22:13:11.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:11.552+0000 7f28ea59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f28ec0f6510 0x7f28ec1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:11.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:11.552+0000 7f28ea59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f28ec0f6510 0x7f28ec1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:34856/0 (socket says 172.21.15.88:34856) 2024-02-19T22:13:11.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:11.552+0000 7f28ea59c700 1 -- 172.21.15.88:0/3222165201 learned_addr learned my addr 172.21.15.88:0/3222165201 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:13:11.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:11.552+0000 7f28ea59c700 1 -- 172.21.15.88:0/3222165201 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28dc005040 con 0x7f28ec0f6510 2024-02-19T22:13:11.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:11.552+0000 7f28ea59c700 1 --2- 172.21.15.88:0/3222165201 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f28ec0f6510 0x7f28ec1067e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f28dc005010 tx=0x7f28dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:11.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:11.552+0000 7f28daffd700 1 -- 172.21.15.88:0/3222165201 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28dc009320 con 0x7f28ec0f6510 2024-02-19T22:13:11.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:11.552+0000 7f28f0921700 1 -- 172.21.15.88:0/3222165201 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28ec106ed0 con 0x7f28ec0f6510 2024-02-19T22:13:11.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:11.553+0000 7f28daffd700 1 -- 172.21.15.88:0/3222165201 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f28dc013380 con 0x7f28ec0f6510 2024-02-19T22:13:11.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:11.553+0000 7f28daffd700 1 -- 172.21.15.88:0/3222165201 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28dc014e50 con 0x7f28ec0f6510 2024-02-19T22:13:11.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:11.553+0000 7f28f0921700 1 -- 172.21.15.88:0/3222165201 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28ec107360 con 0x7f28ec0f6510 2024-02-19T22:13:11.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:11.553+0000 7f28daffd700 1 -- 172.21.15.88:0/3222165201 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f28dc0095f0 con 0x7f28ec0f6510 2024-02-19T22:13:11.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:11.553+0000 7f28f0921700 1 -- 172.21.15.88:0/3222165201 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28ec045ad0 con 0x7f28ec0f6510 2024-02-19T22:13:11.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:11.553+0000 7f28daffd700 1 --2- 172.21.15.88:0/3222165201 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f28d402ff10 0x7f28d40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:11.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:11.553+0000 7f28daffd700 1 -- 172.21.15.88:0/3222165201 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f28dc040e90 con 0x7f28ec0f6510 2024-02-19T22:13:11.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:11.554+0000 7f28e9d9b700 1 --2- 172.21.15.88:0/3222165201 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f28d402ff10 0x7f28d40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:11.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:11.554+0000 7f28e9d9b700 1 --2- 172.21.15.88:0/3222165201 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f28d402ff10 0x7f28d40323d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f28e00096a0 tx=0x7f28e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:11.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:11.558+0000 7f28daffd700 1 -- 172.21.15.88:0/3222165201 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f28dc014420 con 0x7f28ec0f6510 2024-02-19T22:13:11.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:11 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:13:09.647608+0000 mgr.smithi052.ghqrtk (mgr.14162) 104 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:11.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:11.761+0000 7f28f0921700 1 -- 172.21.15.88:0/3222165201 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f28ec108210 con 0x7f28ec0f6510 2024-02-19T22:13:11.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:11.762+0000 7f28daffd700 1 -- 172.21.15.88:0/3222165201 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f28dc010230 con 0x7f28ec0f6510 2024-02-19T22:13:11.763 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:13:11.763 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:13:11.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:11.763+0000 7f28f0921700 1 -- 172.21.15.88:0/3222165201 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f28d402ff10 msgr2=0x7f28d40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:11.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:11.763+0000 7f28f0921700 1 --2- 172.21.15.88:0/3222165201 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f28d402ff10 0x7f28d40323d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f28e00096a0 tx=0x7f28e0006b40).stop 2024-02-19T22:13:11.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:11.763+0000 7f28f0921700 1 -- 172.21.15.88:0/3222165201 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f28ec0f6510 msgr2=0x7f28ec1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:11.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:11.763+0000 7f28f0921700 1 --2- 172.21.15.88:0/3222165201 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f28ec0f6510 0x7f28ec1067e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f28dc005010 tx=0x7f28dc00b040).stop 2024-02-19T22:13:11.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:11.764+0000 7f28f0921700 1 -- 172.21.15.88:0/3222165201 shutdown_connections 2024-02-19T22:13:11.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:11.764+0000 7f28f0921700 1 --2- 172.21.15.88:0/3222165201 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f28d402ff10 0x7f28d40323d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:11.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:11.764+0000 7f28f0921700 1 --2- 172.21.15.88:0/3222165201 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f28ec0f6510 0x7f28ec1067e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:11.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:11.764+0000 7f28f0921700 1 -- 172.21.15.88:0/3222165201 >> 172.21.15.88:0/3222165201 conn(0x7f28ec0f1c70 msgr2=0x7f28ec0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:11.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:11.764+0000 7f28f0921700 1 -- 172.21.15.88:0/3222165201 shutdown_connections 2024-02-19T22:13:11.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:11.764+0000 7f28f0921700 1 -- 172.21.15.88:0/3222165201 wait complete. 2024-02-19T22:13:11.765 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:13:12.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:12 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:13:11.762488+0000 mon.smithi052 (mon.0) 169 : audit [DBG] from='client.? 172.21.15.88:0/3222165201' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:13:13.294 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:13:13.295 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:13:13.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:13 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:13:11.647923+0000 mgr.smithi052.ghqrtk (mgr.14162) 105 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:14.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:14.995+0000 7fbb67938700 1 -- 172.21.15.88:0/2275129841 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbb600f6510 msgr2=0x7fbb600f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:14.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:14.995+0000 7fbb67938700 1 --2- 172.21.15.88:0/2275129841 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbb600f6510 0x7fbb600f6930 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fbb50009fb0 tx=0x7fbb50009b30).stop 2024-02-19T22:13:14.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:14.995+0000 7fbb67938700 1 -- 172.21.15.88:0/2275129841 shutdown_connections 2024-02-19T22:13:14.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:14.995+0000 7fbb67938700 1 --2- 172.21.15.88:0/2275129841 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbb600f6510 0x7fbb600f6930 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:14.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:14.995+0000 7fbb67938700 1 -- 172.21.15.88:0/2275129841 >> 172.21.15.88:0/2275129841 conn(0x7fbb600f1c70 msgr2=0x7fbb600f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:14.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:14.995+0000 7fbb67938700 1 -- 172.21.15.88:0/2275129841 shutdown_connections 2024-02-19T22:13:14.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:14.995+0000 7fbb67938700 1 -- 172.21.15.88:0/2275129841 wait complete. 2024-02-19T22:13:14.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:14.996+0000 7fbb67938700 1 Processor -- start 2024-02-19T22:13:14.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:14.996+0000 7fbb67938700 1 -- start start 2024-02-19T22:13:14.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:14.996+0000 7fbb67938700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbb600f6510 0x7fbb601067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:14.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:14.996+0000 7fbb67938700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb60106cd0 con 0x7fbb600f6510 2024-02-19T22:13:14.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:14.996+0000 7fbb656d4700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbb600f6510 0x7fbb601067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:14.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:14.996+0000 7fbb656d4700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbb600f6510 0x7fbb601067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:45746/0 (socket says 172.21.15.88:45746) 2024-02-19T22:13:14.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:14.996+0000 7fbb656d4700 1 -- 172.21.15.88:0/2170478445 learned_addr learned my addr 172.21.15.88:0/2170478445 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:13:14.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:14.997+0000 7fbb656d4700 1 -- 172.21.15.88:0/2170478445 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb50005040 con 0x7fbb600f6510 2024-02-19T22:13:14.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:14.997+0000 7fbb656d4700 1 --2- 172.21.15.88:0/2170478445 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbb600f6510 0x7fbb601067e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fbb50005010 tx=0x7fbb5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:14.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:14.997+0000 7fbb55ffb700 1 -- 172.21.15.88:0/2170478445 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb50009320 con 0x7fbb600f6510 2024-02-19T22:13:14.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:14.997+0000 7fbb67938700 1 -- 172.21.15.88:0/2170478445 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb60106ed0 con 0x7fbb600f6510 2024-02-19T22:13:15.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:14.997+0000 7fbb55ffb700 1 -- 172.21.15.88:0/2170478445 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbb50013380 con 0x7fbb600f6510 2024-02-19T22:13:15.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:14.997+0000 7fbb55ffb700 1 -- 172.21.15.88:0/2170478445 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb50014e50 con 0x7fbb600f6510 2024-02-19T22:13:15.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:14.997+0000 7fbb67938700 1 -- 172.21.15.88:0/2170478445 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb60107360 con 0x7fbb600f6510 2024-02-19T22:13:15.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:14.998+0000 7fbb55ffb700 1 -- 172.21.15.88:0/2170478445 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbb500095f0 con 0x7fbb600f6510 2024-02-19T22:13:15.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:14.998+0000 7fbb67938700 1 -- 172.21.15.88:0/2170478445 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb60045ad0 con 0x7fbb600f6510 2024-02-19T22:13:15.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:14.998+0000 7fbb55ffb700 1 --2- 172.21.15.88:0/2170478445 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fbb4c02ff10 0x7fbb4c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:15.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:14.998+0000 7fbb55ffb700 1 -- 172.21.15.88:0/2170478445 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbb50040ef0 con 0x7fbb600f6510 2024-02-19T22:13:15.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:14.998+0000 7fbb64ed3700 1 --2- 172.21.15.88:0/2170478445 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fbb4c02ff10 0x7fbb4c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:15.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:14.999+0000 7fbb64ed3700 1 --2- 172.21.15.88:0/2170478445 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fbb4c02ff10 0x7fbb4c0323d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fbb5c0096a0 tx=0x7fbb5c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:15.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:15.003+0000 7fbb55ffb700 1 -- 172.21.15.88:0/2170478445 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbb50014420 con 0x7fbb600f6510 2024-02-19T22:13:15.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:15.207+0000 7fbb67938700 1 -- 172.21.15.88:0/2170478445 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbb601081b0 con 0x7fbb600f6510 2024-02-19T22:13:15.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:15.208+0000 7fbb55ffb700 1 -- 172.21.15.88:0/2170478445 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbb50010230 con 0x7fbb600f6510 2024-02-19T22:13:15.209 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:13:15.209 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:13:15.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:15.210+0000 7fbb67938700 1 -- 172.21.15.88:0/2170478445 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fbb4c02ff10 msgr2=0x7fbb4c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:15.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:15.210+0000 7fbb67938700 1 --2- 172.21.15.88:0/2170478445 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fbb4c02ff10 0x7fbb4c0323d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fbb5c0096a0 tx=0x7fbb5c006b40).stop 2024-02-19T22:13:15.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:15.210+0000 7fbb67938700 1 -- 172.21.15.88:0/2170478445 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbb600f6510 msgr2=0x7fbb601067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:15.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:15.210+0000 7fbb67938700 1 --2- 172.21.15.88:0/2170478445 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbb600f6510 0x7fbb601067e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fbb50005010 tx=0x7fbb5000b040).stop 2024-02-19T22:13:15.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:15.210+0000 7fbb67938700 1 -- 172.21.15.88:0/2170478445 shutdown_connections 2024-02-19T22:13:15.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:15.210+0000 7fbb67938700 1 --2- 172.21.15.88:0/2170478445 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fbb4c02ff10 0x7fbb4c0323d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:15.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:15.210+0000 7fbb67938700 1 --2- 172.21.15.88:0/2170478445 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbb600f6510 0x7fbb601067e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:15.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:15.210+0000 7fbb67938700 1 -- 172.21.15.88:0/2170478445 >> 172.21.15.88:0/2170478445 conn(0x7fbb600f1c70 msgr2=0x7fbb600f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:15.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:15.210+0000 7fbb67938700 1 -- 172.21.15.88:0/2170478445 shutdown_connections 2024-02-19T22:13:15.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:15.210+0000 7fbb67938700 1 -- 172.21.15.88:0/2170478445 wait complete. 2024-02-19T22:13:15.211 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:13:15.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:15 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:13:13.648388+0000 mgr.smithi052.ghqrtk (mgr.14162) 106 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:15.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:15 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:13:15.208482+0000 mon.smithi052 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.88:0/2170478445' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:13:16.844 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:13:16.844 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:13:17.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:17 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:13:15.648868+0000 mgr.smithi052.ghqrtk (mgr.14162) 107 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:18.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:18.783+0000 7f64bba43700 1 -- 172.21.15.88:0/3348421352 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f64b40f5410 msgr2=0x7f64b40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:18.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:18.783+0000 7f64bba43700 1 --2- 172.21.15.88:0/3348421352 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f64b40f5410 0x7f64b40f5830 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f64a4009fb0 tx=0x7f64a4009b30).stop 2024-02-19T22:13:18.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:18.783+0000 7f64bba43700 1 -- 172.21.15.88:0/3348421352 shutdown_connections 2024-02-19T22:13:18.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:18.783+0000 7f64bba43700 1 --2- 172.21.15.88:0/3348421352 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f64b40f5410 0x7f64b40f5830 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:18.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:18.783+0000 7f64bba43700 1 -- 172.21.15.88:0/3348421352 >> 172.21.15.88:0/3348421352 conn(0x7f64b40f0b70 msgr2=0x7f64b40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:18.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:18.783+0000 7f64bba43700 1 -- 172.21.15.88:0/3348421352 shutdown_connections 2024-02-19T22:13:18.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:18.783+0000 7f64bba43700 1 -- 172.21.15.88:0/3348421352 wait complete. 2024-02-19T22:13:18.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:18.784+0000 7f64bba43700 1 Processor -- start 2024-02-19T22:13:18.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:18.784+0000 7f64bba43700 1 -- start start 2024-02-19T22:13:18.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:18.784+0000 7f64bba43700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f64b40f5410 0x7f64b4105720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:18.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:18.784+0000 7f64bba43700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b4105c10 con 0x7f64b40f5410 2024-02-19T22:13:18.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:18.784+0000 7f64b97df700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f64b40f5410 0x7f64b4105720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:18.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:18.784+0000 7f64b97df700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f64b40f5410 0x7f64b4105720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:45762/0 (socket says 172.21.15.88:45762) 2024-02-19T22:13:18.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:18.784+0000 7f64b97df700 1 -- 172.21.15.88:0/3173650533 learned_addr learned my addr 172.21.15.88:0/3173650533 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:13:18.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:18.785+0000 7f64b97df700 1 -- 172.21.15.88:0/3173650533 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a4005040 con 0x7f64b40f5410 2024-02-19T22:13:18.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:18.785+0000 7f64b97df700 1 --2- 172.21.15.88:0/3173650533 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f64b40f5410 0x7f64b4105720 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f64a4005010 tx=0x7f64a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:18.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:18.785+0000 7f64a9ffb700 1 -- 172.21.15.88:0/3173650533 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64a4009320 con 0x7f64b40f5410 2024-02-19T22:13:18.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:18.785+0000 7f64bba43700 1 -- 172.21.15.88:0/3173650533 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64b4105e10 con 0x7f64b40f5410 2024-02-19T22:13:18.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:18.785+0000 7f64a9ffb700 1 -- 172.21.15.88:0/3173650533 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f64a4013380 con 0x7f64b40f5410 2024-02-19T22:13:18.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:18.785+0000 7f64a9ffb700 1 -- 172.21.15.88:0/3173650533 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64a4014e50 con 0x7f64b40f5410 2024-02-19T22:13:18.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:18.785+0000 7f64bba43700 1 -- 172.21.15.88:0/3173650533 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64b41062a0 con 0x7f64b40f5410 2024-02-19T22:13:18.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:18.786+0000 7f64a9ffb700 1 -- 172.21.15.88:0/3173650533 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f64a4014420 con 0x7f64b40f5410 2024-02-19T22:13:18.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:18.786+0000 7f64bba43700 1 -- 172.21.15.88:0/3173650533 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64b4045ad0 con 0x7f64b40f5410 2024-02-19T22:13:18.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:18.786+0000 7f64a9ffb700 1 --2- 172.21.15.88:0/3173650533 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f64a002ff10 0x7f64a00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:18.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:18.786+0000 7f64a9ffb700 1 -- 172.21.15.88:0/3173650533 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f64a4040e60 con 0x7f64b40f5410 2024-02-19T22:13:18.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:18.786+0000 7f64b8fde700 1 --2- 172.21.15.88:0/3173650533 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f64a002ff10 0x7f64a00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:18.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:18.787+0000 7f64b8fde700 1 --2- 172.21.15.88:0/3173650533 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f64a002ff10 0x7f64a00323d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f64b00096a0 tx=0x7f64b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:18.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:18.791+0000 7f64a9ffb700 1 -- 172.21.15.88:0/3173650533 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f64a40095f0 con 0x7f64b40f5410 2024-02-19T22:13:18.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:18.997+0000 7f64bba43700 1 -- 172.21.15.88:0/3173650533 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f64b4107150 con 0x7f64b40f5410 2024-02-19T22:13:19.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:18.997+0000 7f64a9ffb700 1 -- 172.21.15.88:0/3173650533 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f64a401db70 con 0x7f64b40f5410 2024-02-19T22:13:19.000 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:13:19.000 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:13:19.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:18.999+0000 7f64bba43700 1 -- 172.21.15.88:0/3173650533 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f64a002ff10 msgr2=0x7f64a00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:19.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:18.999+0000 7f64bba43700 1 --2- 172.21.15.88:0/3173650533 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f64a002ff10 0x7f64a00323d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f64b00096a0 tx=0x7f64b0006b40).stop 2024-02-19T22:13:19.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:18.999+0000 7f64bba43700 1 -- 172.21.15.88:0/3173650533 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f64b40f5410 msgr2=0x7f64b4105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:19.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:18.999+0000 7f64bba43700 1 --2- 172.21.15.88:0/3173650533 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f64b40f5410 0x7f64b4105720 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f64a4005010 tx=0x7f64a400b040).stop 2024-02-19T22:13:19.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:18.999+0000 7f64bba43700 1 -- 172.21.15.88:0/3173650533 shutdown_connections 2024-02-19T22:13:19.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:18.999+0000 7f64bba43700 1 --2- 172.21.15.88:0/3173650533 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f64a002ff10 0x7f64a00323d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:19.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:18.999+0000 7f64bba43700 1 --2- 172.21.15.88:0/3173650533 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f64b40f5410 0x7f64b4105720 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:19.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:18.999+0000 7f64bba43700 1 -- 172.21.15.88:0/3173650533 >> 172.21.15.88:0/3173650533 conn(0x7f64b40f0b70 msgr2=0x7f64b40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:19.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:18.999+0000 7f64bba43700 1 -- 172.21.15.88:0/3173650533 shutdown_connections 2024-02-19T22:13:19.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:18.999+0000 7f64bba43700 1 -- 172.21.15.88:0/3173650533 wait complete. 2024-02-19T22:13:19.002 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:13:19.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:19 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:13:17.649266+0000 mgr.smithi052.ghqrtk (mgr.14162) 108 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:19.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:19 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:13:18.998971+0000 mon.smithi052 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.88:0/3173650533' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:13:20.573 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:13:20.573 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:13:21.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:21 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:13:19.649506+0000 mgr.smithi052.ghqrtk (mgr.14162) 109 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:22.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:22.416+0000 7fa461961700 1 -- 172.21.15.88:0/881393329 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa45c0f5410 msgr2=0x7fa45c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:22.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:22.416+0000 7fa461961700 1 --2- 172.21.15.88:0/881393329 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa45c0f5410 0x7fa45c0f5830 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fa44c009fb0 tx=0x7fa44c009b30).stop 2024-02-19T22:13:22.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:22.416+0000 7fa461961700 1 -- 172.21.15.88:0/881393329 shutdown_connections 2024-02-19T22:13:22.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:22.416+0000 7fa461961700 1 --2- 172.21.15.88:0/881393329 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa45c0f5410 0x7fa45c0f5830 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:22.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:22.417+0000 7fa461961700 1 -- 172.21.15.88:0/881393329 >> 172.21.15.88:0/881393329 conn(0x7fa45c0f0b70 msgr2=0x7fa45c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:22.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:22.417+0000 7fa461961700 1 -- 172.21.15.88:0/881393329 shutdown_connections 2024-02-19T22:13:22.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:22.417+0000 7fa461961700 1 -- 172.21.15.88:0/881393329 wait complete. 2024-02-19T22:13:22.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:22.417+0000 7fa461961700 1 Processor -- start 2024-02-19T22:13:22.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:22.417+0000 7fa461961700 1 -- start start 2024-02-19T22:13:22.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:22.417+0000 7fa461961700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa45c0f5410 0x7fa45c105720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:22.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:22.417+0000 7fa461961700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45c105c10 con 0x7fa45c0f5410 2024-02-19T22:13:22.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:22.418+0000 7fa45affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa45c0f5410 0x7fa45c105720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:22.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:22.418+0000 7fa45affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa45c0f5410 0x7fa45c105720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:45774/0 (socket says 172.21.15.88:45774) 2024-02-19T22:13:22.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:22.418+0000 7fa45affd700 1 -- 172.21.15.88:0/3257150162 learned_addr learned my addr 172.21.15.88:0/3257150162 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:13:22.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:22.418+0000 7fa45affd700 1 -- 172.21.15.88:0/3257150162 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa44c005040 con 0x7fa45c0f5410 2024-02-19T22:13:22.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:22.418+0000 7fa45affd700 1 --2- 172.21.15.88:0/3257150162 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa45c0f5410 0x7fa45c105720 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fa44c005010 tx=0x7fa44c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:22.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:22.418+0000 7fa4437fe700 1 -- 172.21.15.88:0/3257150162 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa44c009320 con 0x7fa45c0f5410 2024-02-19T22:13:22.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:22.418+0000 7fa4437fe700 1 -- 172.21.15.88:0/3257150162 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa44c013380 con 0x7fa45c0f5410 2024-02-19T22:13:22.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:22.418+0000 7fa461961700 1 -- 172.21.15.88:0/3257150162 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa45c105e10 con 0x7fa45c0f5410 2024-02-19T22:13:22.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:22.418+0000 7fa4437fe700 1 -- 172.21.15.88:0/3257150162 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa44c014e50 con 0x7fa45c0f5410 2024-02-19T22:13:22.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:22.418+0000 7fa461961700 1 -- 172.21.15.88:0/3257150162 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa45c1062a0 con 0x7fa45c0f5410 2024-02-19T22:13:22.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:22.419+0000 7fa4437fe700 1 -- 172.21.15.88:0/3257150162 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa44c014420 con 0x7fa45c0f5410 2024-02-19T22:13:22.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:22.419+0000 7fa461961700 1 -- 172.21.15.88:0/3257150162 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa45c045ad0 con 0x7fa45c0f5410 2024-02-19T22:13:22.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:22.419+0000 7fa4437fe700 1 --2- 172.21.15.88:0/3257150162 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa44402ff10 0x7fa4440323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:22.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:22.419+0000 7fa4437fe700 1 -- 172.21.15.88:0/3257150162 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa44c040f70 con 0x7fa45c0f5410 2024-02-19T22:13:22.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:22.419+0000 7fa45a7fc700 1 --2- 172.21.15.88:0/3257150162 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa44402ff10 0x7fa4440323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:22.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:22.420+0000 7fa45a7fc700 1 --2- 172.21.15.88:0/3257150162 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa44402ff10 0x7fa4440323d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fa4500096a0 tx=0x7fa450006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:22.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:22.424+0000 7fa4437fe700 1 -- 172.21.15.88:0/3257150162 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa44c0095f0 con 0x7fa45c0f5410 2024-02-19T22:13:22.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:22.626+0000 7fa461961700 1 -- 172.21.15.88:0/3257150162 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa45c107150 con 0x7fa45c0f5410 2024-02-19T22:13:22.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:22.626+0000 7fa4437fe700 1 -- 172.21.15.88:0/3257150162 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa44c01db70 con 0x7fa45c0f5410 2024-02-19T22:13:22.628 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:13:22.628 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:13:22.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:22.628+0000 7fa461961700 1 -- 172.21.15.88:0/3257150162 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa44402ff10 msgr2=0x7fa4440323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:22.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:22.628+0000 7fa461961700 1 --2- 172.21.15.88:0/3257150162 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa44402ff10 0x7fa4440323d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fa4500096a0 tx=0x7fa450006b40).stop 2024-02-19T22:13:22.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:22.628+0000 7fa461961700 1 -- 172.21.15.88:0/3257150162 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa45c0f5410 msgr2=0x7fa45c105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:22.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:22.628+0000 7fa461961700 1 --2- 172.21.15.88:0/3257150162 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa45c0f5410 0x7fa45c105720 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fa44c005010 tx=0x7fa44c00b040).stop 2024-02-19T22:13:22.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:22.628+0000 7fa461961700 1 -- 172.21.15.88:0/3257150162 shutdown_connections 2024-02-19T22:13:22.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:22.628+0000 7fa461961700 1 --2- 172.21.15.88:0/3257150162 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa44402ff10 0x7fa4440323d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:22.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:22.628+0000 7fa461961700 1 --2- 172.21.15.88:0/3257150162 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa45c0f5410 0x7fa45c105720 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:22.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:22.628+0000 7fa461961700 1 -- 172.21.15.88:0/3257150162 >> 172.21.15.88:0/3257150162 conn(0x7fa45c0f0b70 msgr2=0x7fa45c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:22.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:22.628+0000 7fa461961700 1 -- 172.21.15.88:0/3257150162 shutdown_connections 2024-02-19T22:13:22.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:22.628+0000 7fa461961700 1 -- 172.21.15.88:0/3257150162 wait complete. 2024-02-19T22:13:22.631 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:13:23.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:23 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:13:21.649802+0000 mgr.smithi052.ghqrtk (mgr.14162) 110 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:23.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:23 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:13:22.627763+0000 mon.smithi052 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.88:0/3257150162' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:13:24.277 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:13:24.277 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:13:25.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:25 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:13:23.650191+0000 mgr.smithi052.ghqrtk (mgr.14162) 111 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:26.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:26.038+0000 7f67f25b5700 1 -- 172.21.15.88:0/3436147552 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f67ec0f62a0 msgr2=0x7f67ec0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:26.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:26.038+0000 7f67f25b5700 1 --2- 172.21.15.88:0/3436147552 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f67ec0f62a0 0x7f67ec0f66c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f67e8009fb0 tx=0x7f67e8009b30).stop 2024-02-19T22:13:26.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:26.038+0000 7f67f25b5700 1 -- 172.21.15.88:0/3436147552 shutdown_connections 2024-02-19T22:13:26.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:26.038+0000 7f67f25b5700 1 --2- 172.21.15.88:0/3436147552 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f67ec0f62a0 0x7f67ec0f66c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:26.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:26.038+0000 7f67f25b5700 1 -- 172.21.15.88:0/3436147552 >> 172.21.15.88:0/3436147552 conn(0x7f67ec0f1aa0 msgr2=0x7f67ec0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:26.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:26.038+0000 7f67f25b5700 1 -- 172.21.15.88:0/3436147552 shutdown_connections 2024-02-19T22:13:26.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:26.038+0000 7f67f25b5700 1 -- 172.21.15.88:0/3436147552 wait complete. 2024-02-19T22:13:26.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:26.038+0000 7f67f25b5700 1 Processor -- start 2024-02-19T22:13:26.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:26.038+0000 7f67f25b5700 1 -- start start 2024-02-19T22:13:26.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:26.038+0000 7f67f25b5700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f67ec0f62a0 0x7f67ec06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:26.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:26.038+0000 7f67f25b5700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67ec069e20 con 0x7f67ec0f62a0 2024-02-19T22:13:26.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:26.039+0000 7f67f15b3700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f67ec0f62a0 0x7f67ec06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:26.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:26.039+0000 7f67f15b3700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f67ec0f62a0 0x7f67ec06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:60764/0 (socket says 172.21.15.88:60764) 2024-02-19T22:13:26.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:26.039+0000 7f67f15b3700 1 -- 172.21.15.88:0/1824067102 learned_addr learned my addr 172.21.15.88:0/1824067102 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:13:26.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:26.039+0000 7f67f15b3700 1 -- 172.21.15.88:0/1824067102 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67e8005040 con 0x7f67ec0f62a0 2024-02-19T22:13:26.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:26.039+0000 7f67f15b3700 1 --2- 172.21.15.88:0/1824067102 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f67ec0f62a0 0x7f67ec06b9a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f67e80089f0 tx=0x7f67e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:26.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:26.039+0000 7f67e1ffb700 1 -- 172.21.15.88:0/1824067102 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67e8009320 con 0x7f67ec0f62a0 2024-02-19T22:13:26.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:26.039+0000 7f67f25b5700 1 -- 172.21.15.88:0/1824067102 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67ec069fc0 con 0x7f67ec0f62a0 2024-02-19T22:13:26.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:26.039+0000 7f67e1ffb700 1 -- 172.21.15.88:0/1824067102 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f67e8013380 con 0x7f67ec0f62a0 2024-02-19T22:13:26.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:26.039+0000 7f67e1ffb700 1 -- 172.21.15.88:0/1824067102 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67e8014e20 con 0x7f67ec0f62a0 2024-02-19T22:13:26.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:26.039+0000 7f67f25b5700 1 -- 172.21.15.88:0/1824067102 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67ec06a450 con 0x7f67ec0f62a0 2024-02-19T22:13:26.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:26.040+0000 7f67e1ffb700 1 -- 172.21.15.88:0/1824067102 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f67e80263a0 con 0x7f67ec0f62a0 2024-02-19T22:13:26.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:26.040+0000 7f67f25b5700 1 -- 172.21.15.88:0/1824067102 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67ec059100 con 0x7f67ec0f62a0 2024-02-19T22:13:26.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:26.040+0000 7f67e1ffb700 1 --2- 172.21.15.88:0/1824067102 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f67d802ff10 0x7f67d80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:26.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:26.040+0000 7f67e1ffb700 1 -- 172.21.15.88:0/1824067102 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f67e8041260 con 0x7f67ec0f62a0 2024-02-19T22:13:26.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:26.041+0000 7f67f0db2700 1 --2- 172.21.15.88:0/1824067102 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f67d802ff10 0x7f67d80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:26.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:26.041+0000 7f67f0db2700 1 --2- 172.21.15.88:0/1824067102 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f67d802ff10 0x7f67d80323d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f67dc0096a0 tx=0x7f67dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:26.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:26.045+0000 7f67e1ffb700 1 -- 172.21.15.88:0/1824067102 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f67e80126f0 con 0x7f67ec0f62a0 2024-02-19T22:13:26.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:26.247+0000 7f67f25b5700 1 -- 172.21.15.88:0/1824067102 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f67ec045ad0 con 0x7f67ec0f62a0 2024-02-19T22:13:26.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:26.248+0000 7f67e1ffb700 1 -- 172.21.15.88:0/1824067102 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f67e80126f0 con 0x7f67ec0f62a0 2024-02-19T22:13:26.250 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:13:26.250 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:13:26.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:26.250+0000 7f67f25b5700 1 -- 172.21.15.88:0/1824067102 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f67d802ff10 msgr2=0x7f67d80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:26.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:26.250+0000 7f67f25b5700 1 --2- 172.21.15.88:0/1824067102 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f67d802ff10 0x7f67d80323d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f67dc0096a0 tx=0x7f67dc006b40).stop 2024-02-19T22:13:26.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:26.250+0000 7f67f25b5700 1 -- 172.21.15.88:0/1824067102 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f67ec0f62a0 msgr2=0x7f67ec06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:26.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:26.250+0000 7f67f25b5700 1 --2- 172.21.15.88:0/1824067102 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f67ec0f62a0 0x7f67ec06b9a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f67e80089f0 tx=0x7f67e800b040).stop 2024-02-19T22:13:26.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:26.250+0000 7f67f25b5700 1 -- 172.21.15.88:0/1824067102 shutdown_connections 2024-02-19T22:13:26.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:26.250+0000 7f67f25b5700 1 --2- 172.21.15.88:0/1824067102 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f67d802ff10 0x7f67d80323d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:26.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:26.250+0000 7f67f25b5700 1 --2- 172.21.15.88:0/1824067102 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f67ec0f62a0 0x7f67ec06b9a0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:26.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:26.250+0000 7f67f25b5700 1 -- 172.21.15.88:0/1824067102 >> 172.21.15.88:0/1824067102 conn(0x7f67ec0f1aa0 msgr2=0x7f67ec0f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:26.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:26.250+0000 7f67f25b5700 1 -- 172.21.15.88:0/1824067102 shutdown_connections 2024-02-19T22:13:26.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:26.250+0000 7f67f25b5700 1 -- 172.21.15.88:0/1824067102 wait complete. 2024-02-19T22:13:26.253 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:13:27.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:27 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:13:25.650589+0000 mgr.smithi052.ghqrtk (mgr.14162) 112 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:27.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:27 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:13:26.249609+0000 mon.smithi052 (mon.0) 173 : audit [DBG] from='client.? 172.21.15.88:0/1824067102' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:13:27.845 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:13:27.846 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:13:29.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:29 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:13:27.650957+0000 mgr.smithi052.ghqrtk (mgr.14162) 113 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:29.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:29.616+0000 7ff7d5647700 1 -- 172.21.15.88:0/3387162228 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff7d00f62a0 msgr2=0x7ff7d00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:29.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:29.616+0000 7ff7d5647700 1 --2- 172.21.15.88:0/3387162228 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff7d00f62a0 0x7ff7d00f66c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7ff7c0009fb0 tx=0x7ff7c0009b30).stop 2024-02-19T22:13:29.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:29.616+0000 7ff7d5647700 1 -- 172.21.15.88:0/3387162228 shutdown_connections 2024-02-19T22:13:29.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:29.616+0000 7ff7d5647700 1 --2- 172.21.15.88:0/3387162228 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff7d00f62a0 0x7ff7d00f66c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:29.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:29.616+0000 7ff7d5647700 1 -- 172.21.15.88:0/3387162228 >> 172.21.15.88:0/3387162228 conn(0x7ff7d00f1aa0 msgr2=0x7ff7d00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:29.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:29.616+0000 7ff7d5647700 1 -- 172.21.15.88:0/3387162228 shutdown_connections 2024-02-19T22:13:29.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:29.616+0000 7ff7d5647700 1 -- 172.21.15.88:0/3387162228 wait complete. 2024-02-19T22:13:29.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:29.616+0000 7ff7d5647700 1 Processor -- start 2024-02-19T22:13:29.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:29.617+0000 7ff7d5647700 1 -- start start 2024-02-19T22:13:29.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:29.617+0000 7ff7d5647700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff7d00f62a0 0x7ff7d01065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:29.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:29.617+0000 7ff7d5647700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d0106ac0 con 0x7ff7d00f62a0 2024-02-19T22:13:29.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:29.617+0000 7ff7cffff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff7d00f62a0 0x7ff7d01065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:29.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:29.617+0000 7ff7cffff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff7d00f62a0 0x7ff7d01065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:60774/0 (socket says 172.21.15.88:60774) 2024-02-19T22:13:29.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:29.617+0000 7ff7cffff700 1 -- 172.21.15.88:0/2225463260 learned_addr learned my addr 172.21.15.88:0/2225463260 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:13:29.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:29.617+0000 7ff7cffff700 1 -- 172.21.15.88:0/2225463260 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7c0005040 con 0x7ff7d00f62a0 2024-02-19T22:13:29.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:29.618+0000 7ff7cffff700 1 --2- 172.21.15.88:0/2225463260 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff7d00f62a0 0x7ff7d01065d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7ff7c00089f0 tx=0x7ff7c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:29.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:29.618+0000 7ff7ccff9700 1 -- 172.21.15.88:0/2225463260 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7c0009320 con 0x7ff7d00f62a0 2024-02-19T22:13:29.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:29.618+0000 7ff7d5647700 1 -- 172.21.15.88:0/2225463260 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7d0106cc0 con 0x7ff7d00f62a0 2024-02-19T22:13:29.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:29.618+0000 7ff7ccff9700 1 -- 172.21.15.88:0/2225463260 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff7c0013380 con 0x7ff7d00f62a0 2024-02-19T22:13:29.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:29.618+0000 7ff7ccff9700 1 -- 172.21.15.88:0/2225463260 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7c0014e20 con 0x7ff7d00f62a0 2024-02-19T22:13:29.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:29.618+0000 7ff7d5647700 1 -- 172.21.15.88:0/2225463260 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7d0107150 con 0x7ff7d00f62a0 2024-02-19T22:13:29.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:29.619+0000 7ff7ccff9700 1 -- 172.21.15.88:0/2225463260 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff7c00263a0 con 0x7ff7d00f62a0 2024-02-19T22:13:29.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:29.619+0000 7ff7ccff9700 1 --2- 172.21.15.88:0/2225463260 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7ff7b802ff10 0x7ff7b80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:29.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:29.619+0000 7ff7ccff9700 1 -- 172.21.15.88:0/2225463260 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff7c0040f90 con 0x7ff7d00f62a0 2024-02-19T22:13:29.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:29.619+0000 7ff7d5647700 1 -- 172.21.15.88:0/2225463260 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7d0045ad0 con 0x7ff7d00f62a0 2024-02-19T22:13:29.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:29.619+0000 7ff7cf7fe700 1 --2- 172.21.15.88:0/2225463260 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7ff7b802ff10 0x7ff7b80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:29.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:29.619+0000 7ff7cf7fe700 1 --2- 172.21.15.88:0/2225463260 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7ff7b802ff10 0x7ff7b80323d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7ff7c40096a0 tx=0x7ff7c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:29.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:29.624+0000 7ff7ccff9700 1 -- 172.21.15.88:0/2225463260 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff7c0012920 con 0x7ff7d00f62a0 2024-02-19T22:13:29.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:29.827+0000 7ff7d5647700 1 -- 172.21.15.88:0/2225463260 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff7d0107ea0 con 0x7ff7d00f62a0 2024-02-19T22:13:29.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:29.828+0000 7ff7ccff9700 1 -- 172.21.15.88:0/2225463260 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff7c002b020 con 0x7ff7d00f62a0 2024-02-19T22:13:29.830 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:13:29.830 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:13:29.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:29.829+0000 7ff7d5647700 1 -- 172.21.15.88:0/2225463260 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7ff7b802ff10 msgr2=0x7ff7b80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:29.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:29.829+0000 7ff7d5647700 1 --2- 172.21.15.88:0/2225463260 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7ff7b802ff10 0x7ff7b80323d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7ff7c40096a0 tx=0x7ff7c4006b40).stop 2024-02-19T22:13:29.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:29.829+0000 7ff7d5647700 1 -- 172.21.15.88:0/2225463260 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff7d00f62a0 msgr2=0x7ff7d01065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:29.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:29.829+0000 7ff7d5647700 1 --2- 172.21.15.88:0/2225463260 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff7d00f62a0 0x7ff7d01065d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7ff7c00089f0 tx=0x7ff7c000b040).stop 2024-02-19T22:13:29.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:29.829+0000 7ff7d5647700 1 -- 172.21.15.88:0/2225463260 shutdown_connections 2024-02-19T22:13:29.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:29.829+0000 7ff7d5647700 1 --2- 172.21.15.88:0/2225463260 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7ff7b802ff10 0x7ff7b80323d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:29.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:29.829+0000 7ff7d5647700 1 --2- 172.21.15.88:0/2225463260 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff7d00f62a0 0x7ff7d01065d0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:29.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:29.829+0000 7ff7d5647700 1 -- 172.21.15.88:0/2225463260 >> 172.21.15.88:0/2225463260 conn(0x7ff7d00f1aa0 msgr2=0x7ff7d00f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:29.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:29.829+0000 7ff7d5647700 1 -- 172.21.15.88:0/2225463260 shutdown_connections 2024-02-19T22:13:29.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:29.829+0000 7ff7d5647700 1 -- 172.21.15.88:0/2225463260 wait complete. 2024-02-19T22:13:29.832 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:13:30.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:30 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:13:29.829073+0000 mon.smithi052 (mon.0) 174 : audit [DBG] from='client.? 172.21.15.88:0/2225463260' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:13:31.441 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:13:31.442 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:13:31.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:31 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:13:29.651332+0000 mgr.smithi052.ghqrtk (mgr.14162) 114 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:33.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:33.288+0000 7fb417abc700 1 -- 172.21.15.88:0/2881164480 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb4100f6510 msgr2=0x7fb4100f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:33.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:33.288+0000 7fb417abc700 1 --2- 172.21.15.88:0/2881164480 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb4100f6510 0x7fb4100f6930 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fb404009fb0 tx=0x7fb404009b30).stop 2024-02-19T22:13:33.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:33.288+0000 7fb417abc700 1 -- 172.21.15.88:0/2881164480 shutdown_connections 2024-02-19T22:13:33.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:33.288+0000 7fb417abc700 1 --2- 172.21.15.88:0/2881164480 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb4100f6510 0x7fb4100f6930 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:33.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:33.288+0000 7fb417abc700 1 -- 172.21.15.88:0/2881164480 >> 172.21.15.88:0/2881164480 conn(0x7fb4100f1c70 msgr2=0x7fb4100f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:33.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:33.288+0000 7fb417abc700 1 -- 172.21.15.88:0/2881164480 shutdown_connections 2024-02-19T22:13:33.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:33.288+0000 7fb417abc700 1 -- 172.21.15.88:0/2881164480 wait complete. 2024-02-19T22:13:33.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:33.289+0000 7fb417abc700 1 Processor -- start 2024-02-19T22:13:33.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:33.289+0000 7fb417abc700 1 -- start start 2024-02-19T22:13:33.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:33.289+0000 7fb417abc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb410106900 0x7fb410106d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:33.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:33.289+0000 7fb417abc700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb410107210 con 0x7fb410106900 2024-02-19T22:13:33.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:33.289+0000 7fb415858700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb410106900 0x7fb410106d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:33.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:33.289+0000 7fb415858700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb410106900 0x7fb410106d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:60790/0 (socket says 172.21.15.88:60790) 2024-02-19T22:13:33.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:33.289+0000 7fb415858700 1 -- 172.21.15.88:0/3324649395 learned_addr learned my addr 172.21.15.88:0/3324649395 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:13:33.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:33.289+0000 7fb415858700 1 -- 172.21.15.88:0/3324649395 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb404005040 con 0x7fb410106900 2024-02-19T22:13:33.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:33.289+0000 7fb415858700 1 --2- 172.21.15.88:0/3324649395 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb410106900 0x7fb410106d20 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fb4100f7610 tx=0x7fb404009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:33.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:33.290+0000 7fb4027fc700 1 -- 172.21.15.88:0/3324649395 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb404013380 con 0x7fb410106900 2024-02-19T22:13:33.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:33.290+0000 7fb4027fc700 1 -- 172.21.15.88:0/3324649395 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb404013790 con 0x7fb410106900 2024-02-19T22:13:33.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:33.290+0000 7fb417abc700 1 -- 172.21.15.88:0/3324649395 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb410107410 con 0x7fb410106900 2024-02-19T22:13:33.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:33.290+0000 7fb4027fc700 1 -- 172.21.15.88:0/3324649395 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb40401ee70 con 0x7fb410106900 2024-02-19T22:13:33.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:33.290+0000 7fb417abc700 1 -- 172.21.15.88:0/3324649395 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb410107f60 con 0x7fb410106900 2024-02-19T22:13:33.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:33.290+0000 7fb4027fc700 1 -- 172.21.15.88:0/3324649395 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb404025060 con 0x7fb410106900 2024-02-19T22:13:33.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:33.290+0000 7fb417abc700 1 -- 172.21.15.88:0/3324649395 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb410059100 con 0x7fb410106900 2024-02-19T22:13:33.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:33.291+0000 7fb4027fc700 1 --2- 172.21.15.88:0/3324649395 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fb3fc02ff60 0x7fb3fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:33.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:33.291+0000 7fb4027fc700 1 -- 172.21.15.88:0/3324649395 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb404040e80 con 0x7fb410106900 2024-02-19T22:13:33.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:33.291+0000 7fb415057700 1 --2- 172.21.15.88:0/3324649395 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fb3fc02ff60 0x7fb3fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:33.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:33.291+0000 7fb415057700 1 --2- 172.21.15.88:0/3324649395 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fb3fc02ff60 0x7fb3fc032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fb40c0096a0 tx=0x7fb40c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:33.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:33.295+0000 7fb4027fc700 1 -- 172.21.15.88:0/3324649395 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb40401c950 con 0x7fb410106900 2024-02-19T22:13:33.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:33.498+0000 7fb417abc700 1 -- 172.21.15.88:0/3324649395 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb410108720 con 0x7fb410106900 2024-02-19T22:13:33.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:33.499+0000 7fb4027fc700 1 -- 172.21.15.88:0/3324649395 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb40401cb60 con 0x7fb410106900 2024-02-19T22:13:33.501 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:13:33.501 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:13:33.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:33.501+0000 7fb417abc700 1 -- 172.21.15.88:0/3324649395 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fb3fc02ff60 msgr2=0x7fb3fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:33.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:33.501+0000 7fb417abc700 1 --2- 172.21.15.88:0/3324649395 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fb3fc02ff60 0x7fb3fc032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fb40c0096a0 tx=0x7fb40c006b40).stop 2024-02-19T22:13:33.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:33.501+0000 7fb417abc700 1 -- 172.21.15.88:0/3324649395 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb410106900 msgr2=0x7fb410106d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:33.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:33.501+0000 7fb417abc700 1 --2- 172.21.15.88:0/3324649395 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb410106900 0x7fb410106d20 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fb4100f7610 tx=0x7fb404009b30).stop 2024-02-19T22:13:33.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:33.501+0000 7fb417abc700 1 -- 172.21.15.88:0/3324649395 shutdown_connections 2024-02-19T22:13:33.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:33.501+0000 7fb417abc700 1 --2- 172.21.15.88:0/3324649395 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fb3fc02ff60 0x7fb3fc032420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:33.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:33.501+0000 7fb417abc700 1 --2- 172.21.15.88:0/3324649395 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb410106900 0x7fb410106d20 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:33.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:33.501+0000 7fb417abc700 1 -- 172.21.15.88:0/3324649395 >> 172.21.15.88:0/3324649395 conn(0x7fb4100f1c70 msgr2=0x7fb4100f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:33.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:33.501+0000 7fb417abc700 1 -- 172.21.15.88:0/3324649395 shutdown_connections 2024-02-19T22:13:33.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:33.501+0000 7fb417abc700 1 -- 172.21.15.88:0/3324649395 wait complete. 2024-02-19T22:13:33.503 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:13:33.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:33 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:13:31.651681+0000 mgr.smithi052.ghqrtk (mgr.14162) 115 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:34.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:34 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:13:33.500571+0000 mon.smithi052 (mon.0) 175 : audit [DBG] from='client.? 172.21.15.88:0/3324649395' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:13:35.110 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:13:35.110 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:13:35.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:35 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:13:33.652053+0000 mgr.smithi052.ghqrtk (mgr.14162) 116 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:37.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:37.040+0000 7f1a4b198700 1 -- 172.21.15.88:0/1178029410 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a440f6510 msgr2=0x7f1a440f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:37.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:37.040+0000 7f1a4b198700 1 --2- 172.21.15.88:0/1178029410 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a440f6510 0x7f1a440f6930 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f1a38009fb0 tx=0x7f1a38009b30).stop 2024-02-19T22:13:37.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:37.040+0000 7f1a4b198700 1 -- 172.21.15.88:0/1178029410 shutdown_connections 2024-02-19T22:13:37.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:37.040+0000 7f1a4b198700 1 --2- 172.21.15.88:0/1178029410 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a440f6510 0x7f1a440f6930 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:37.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:37.040+0000 7f1a4b198700 1 -- 172.21.15.88:0/1178029410 >> 172.21.15.88:0/1178029410 conn(0x7f1a440f1c70 msgr2=0x7f1a440f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:37.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:37.040+0000 7f1a4b198700 1 -- 172.21.15.88:0/1178029410 shutdown_connections 2024-02-19T22:13:37.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:37.040+0000 7f1a4b198700 1 -- 172.21.15.88:0/1178029410 wait complete. 2024-02-19T22:13:37.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:37.041+0000 7f1a4b198700 1 Processor -- start 2024-02-19T22:13:37.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:37.041+0000 7f1a4b198700 1 -- start start 2024-02-19T22:13:37.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:37.041+0000 7f1a4b198700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a440f6510 0x7f1a441067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:37.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:37.041+0000 7f1a4b198700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a44106cd0 con 0x7f1a440f6510 2024-02-19T22:13:37.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:37.042+0000 7f1a48f34700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a440f6510 0x7f1a441067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:37.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:37.042+0000 7f1a48f34700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a440f6510 0x7f1a441067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:45232/0 (socket says 172.21.15.88:45232) 2024-02-19T22:13:37.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:37.042+0000 7f1a48f34700 1 -- 172.21.15.88:0/2834249758 learned_addr learned my addr 172.21.15.88:0/2834249758 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:13:37.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:37.042+0000 7f1a48f34700 1 -- 172.21.15.88:0/2834249758 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a38005040 con 0x7f1a440f6510 2024-02-19T22:13:37.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:37.042+0000 7f1a48f34700 1 --2- 172.21.15.88:0/2834249758 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a440f6510 0x7f1a441067e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f1a38005010 tx=0x7f1a3800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:37.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:37.042+0000 7f1a417fa700 1 -- 172.21.15.88:0/2834249758 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a38009320 con 0x7f1a440f6510 2024-02-19T22:13:37.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:37.042+0000 7f1a4b198700 1 -- 172.21.15.88:0/2834249758 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a44106ed0 con 0x7f1a440f6510 2024-02-19T22:13:37.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:37.042+0000 7f1a417fa700 1 -- 172.21.15.88:0/2834249758 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1a38013380 con 0x7f1a440f6510 2024-02-19T22:13:37.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:37.042+0000 7f1a417fa700 1 -- 172.21.15.88:0/2834249758 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a38014e50 con 0x7f1a440f6510 2024-02-19T22:13:37.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:37.042+0000 7f1a4b198700 1 -- 172.21.15.88:0/2834249758 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a44107360 con 0x7f1a440f6510 2024-02-19T22:13:37.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:37.043+0000 7f1a417fa700 1 -- 172.21.15.88:0/2834249758 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1a380095f0 con 0x7f1a440f6510 2024-02-19T22:13:37.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:37.043+0000 7f1a4b198700 1 -- 172.21.15.88:0/2834249758 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a44045ad0 con 0x7f1a440f6510 2024-02-19T22:13:37.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:37.043+0000 7f1a417fa700 1 --2- 172.21.15.88:0/2834249758 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f1a3402ff10 0x7f1a340323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:37.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:37.043+0000 7f1a417fa700 1 -- 172.21.15.88:0/2834249758 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1a38040f80 con 0x7f1a440f6510 2024-02-19T22:13:37.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:37.043+0000 7f1a43fff700 1 --2- 172.21.15.88:0/2834249758 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f1a3402ff10 0x7f1a340323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:37.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:37.044+0000 7f1a43fff700 1 --2- 172.21.15.88:0/2834249758 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f1a3402ff10 0x7f1a340323d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f1a300096a0 tx=0x7f1a30006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:37.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:37.048+0000 7f1a417fa700 1 -- 172.21.15.88:0/2834249758 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1a38014420 con 0x7f1a440f6510 2024-02-19T22:13:37.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:37.253+0000 7f1a4b198700 1 -- 172.21.15.88:0/2834249758 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1a44108210 con 0x7f1a440f6510 2024-02-19T22:13:37.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:37.253+0000 7f1a417fa700 1 -- 172.21.15.88:0/2834249758 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1a38010230 con 0x7f1a440f6510 2024-02-19T22:13:37.255 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:13:37.256 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:13:37.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:37.255+0000 7f1a4b198700 1 -- 172.21.15.88:0/2834249758 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f1a3402ff10 msgr2=0x7f1a340323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:37.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:37.255+0000 7f1a4b198700 1 --2- 172.21.15.88:0/2834249758 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f1a3402ff10 0x7f1a340323d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f1a300096a0 tx=0x7f1a30006b40).stop 2024-02-19T22:13:37.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:37.255+0000 7f1a4b198700 1 -- 172.21.15.88:0/2834249758 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a440f6510 msgr2=0x7f1a441067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:37.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:37.255+0000 7f1a4b198700 1 --2- 172.21.15.88:0/2834249758 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a440f6510 0x7f1a441067e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f1a38005010 tx=0x7f1a3800b040).stop 2024-02-19T22:13:37.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:37.255+0000 7f1a4b198700 1 -- 172.21.15.88:0/2834249758 shutdown_connections 2024-02-19T22:13:37.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:37.255+0000 7f1a4b198700 1 --2- 172.21.15.88:0/2834249758 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f1a3402ff10 0x7f1a340323d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:37.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:37.255+0000 7f1a4b198700 1 --2- 172.21.15.88:0/2834249758 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a440f6510 0x7f1a441067e0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:37.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:37.255+0000 7f1a4b198700 1 -- 172.21.15.88:0/2834249758 >> 172.21.15.88:0/2834249758 conn(0x7f1a440f1c70 msgr2=0x7f1a440f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:37.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:37.255+0000 7f1a4b198700 1 -- 172.21.15.88:0/2834249758 shutdown_connections 2024-02-19T22:13:37.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:37.255+0000 7f1a4b198700 1 -- 172.21.15.88:0/2834249758 wait complete. 2024-02-19T22:13:37.258 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:13:37.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:37 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:13:35.652308+0000 mgr.smithi052.ghqrtk (mgr.14162) 117 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:37.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:37 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:13:37.255062+0000 mon.smithi052 (mon.0) 176 : audit [DBG] from='client.? 172.21.15.88:0/2834249758' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:13:38.914 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:13:38.915 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:13:39.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:13:37.652648+0000 mgr.smithi052.ghqrtk (mgr.14162) 118 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:40.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:40.713+0000 7f8594bf4700 1 -- 172.21.15.88:0/1509337051 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f85900f5410 msgr2=0x7f85900f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:40.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:40.713+0000 7f8594bf4700 1 --2- 172.21.15.88:0/1509337051 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f85900f5410 0x7f85900f5830 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f8584009fb0 tx=0x7f8584009b30).stop 2024-02-19T22:13:40.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:40.713+0000 7f8594bf4700 1 -- 172.21.15.88:0/1509337051 shutdown_connections 2024-02-19T22:13:40.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:40.713+0000 7f8594bf4700 1 --2- 172.21.15.88:0/1509337051 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f85900f5410 0x7f85900f5830 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:40.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:40.713+0000 7f8594bf4700 1 -- 172.21.15.88:0/1509337051 >> 172.21.15.88:0/1509337051 conn(0x7f85900f0b70 msgr2=0x7f85900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:40.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:40.713+0000 7f8594bf4700 1 -- 172.21.15.88:0/1509337051 shutdown_connections 2024-02-19T22:13:40.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:40.713+0000 7f8594bf4700 1 -- 172.21.15.88:0/1509337051 wait complete. 2024-02-19T22:13:40.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:40.713+0000 7f8594bf4700 1 Processor -- start 2024-02-19T22:13:40.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:40.714+0000 7f8594bf4700 1 -- start start 2024-02-19T22:13:40.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:40.714+0000 7f8594bf4700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f85900f5410 0x7f85901067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:40.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:40.714+0000 7f8594bf4700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8590106cd0 con 0x7f85900f5410 2024-02-19T22:13:40.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:40.714+0000 7f858e59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f85900f5410 0x7f85901067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:40.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:40.714+0000 7f858e59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f85900f5410 0x7f85901067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:45252/0 (socket says 172.21.15.88:45252) 2024-02-19T22:13:40.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:40.714+0000 7f858e59c700 1 -- 172.21.15.88:0/928827573 learned_addr learned my addr 172.21.15.88:0/928827573 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:13:40.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:40.714+0000 7f858e59c700 1 -- 172.21.15.88:0/928827573 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8584005040 con 0x7f85900f5410 2024-02-19T22:13:40.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:40.714+0000 7f858e59c700 1 --2- 172.21.15.88:0/928827573 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f85900f5410 0x7f85901067e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f8584005010 tx=0x7f858400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:40.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:40.714+0000 7f857effd700 1 -- 172.21.15.88:0/928827573 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8584009320 con 0x7f85900f5410 2024-02-19T22:13:40.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:40.715+0000 7f8594bf4700 1 -- 172.21.15.88:0/928827573 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8590106ed0 con 0x7f85900f5410 2024-02-19T22:13:40.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:40.715+0000 7f857effd700 1 -- 172.21.15.88:0/928827573 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8584013380 con 0x7f85900f5410 2024-02-19T22:13:40.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:40.715+0000 7f857effd700 1 -- 172.21.15.88:0/928827573 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8584014e50 con 0x7f85900f5410 2024-02-19T22:13:40.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:40.715+0000 7f8594bf4700 1 -- 172.21.15.88:0/928827573 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8590107360 con 0x7f85900f5410 2024-02-19T22:13:40.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:40.715+0000 7f857effd700 1 -- 172.21.15.88:0/928827573 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f85840095f0 con 0x7f85900f5410 2024-02-19T22:13:40.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:40.715+0000 7f8594bf4700 1 -- 172.21.15.88:0/928827573 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8590045ad0 con 0x7f85900f5410 2024-02-19T22:13:40.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:40.715+0000 7f857effd700 1 --2- 172.21.15.88:0/928827573 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f857802ff10 0x7f85780323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:40.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:40.715+0000 7f857effd700 1 -- 172.21.15.88:0/928827573 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8584040fa0 con 0x7f85900f5410 2024-02-19T22:13:40.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:40.716+0000 7f858dd9b700 1 --2- 172.21.15.88:0/928827573 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f857802ff10 0x7f85780323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:40.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:40.716+0000 7f858dd9b700 1 --2- 172.21.15.88:0/928827573 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f857802ff10 0x7f85780323d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f85800096a0 tx=0x7f8580006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:40.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:40.720+0000 7f857effd700 1 -- 172.21.15.88:0/928827573 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8584014420 con 0x7f85900f5410 2024-02-19T22:13:40.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:40.924+0000 7f8594bf4700 1 -- 172.21.15.88:0/928827573 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f85900fafa0 con 0x7f85900f5410 2024-02-19T22:13:40.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:40.924+0000 7f857effd700 1 -- 172.21.15.88:0/928827573 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8584010230 con 0x7f85900f5410 2024-02-19T22:13:40.926 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:13:40.926 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:13:40.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:40.926+0000 7f8594bf4700 1 -- 172.21.15.88:0/928827573 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f857802ff10 msgr2=0x7f85780323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:40.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:40.926+0000 7f8594bf4700 1 --2- 172.21.15.88:0/928827573 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f857802ff10 0x7f85780323d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f85800096a0 tx=0x7f8580006b40).stop 2024-02-19T22:13:40.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:40.926+0000 7f8594bf4700 1 -- 172.21.15.88:0/928827573 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f85900f5410 msgr2=0x7f85901067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:40.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:40.926+0000 7f8594bf4700 1 --2- 172.21.15.88:0/928827573 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f85900f5410 0x7f85901067e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f8584005010 tx=0x7f858400b040).stop 2024-02-19T22:13:40.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:40.926+0000 7f8594bf4700 1 -- 172.21.15.88:0/928827573 shutdown_connections 2024-02-19T22:13:40.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:40.926+0000 7f8594bf4700 1 --2- 172.21.15.88:0/928827573 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f857802ff10 0x7f85780323d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:40.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:40.926+0000 7f8594bf4700 1 --2- 172.21.15.88:0/928827573 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f85900f5410 0x7f85901067e0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:40.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:40.926+0000 7f8594bf4700 1 -- 172.21.15.88:0/928827573 >> 172.21.15.88:0/928827573 conn(0x7f85900f0b70 msgr2=0x7f859005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:40.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:40.926+0000 7f8594bf4700 1 -- 172.21.15.88:0/928827573 shutdown_connections 2024-02-19T22:13:40.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:40.926+0000 7f8594bf4700 1 -- 172.21.15.88:0/928827573 wait complete. 2024-02-19T22:13:40.929 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:13:41.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:41 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:13:39.652941+0000 mgr.smithi052.ghqrtk (mgr.14162) 119 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:41.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:41 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:13:40.925902+0000 mon.smithi052 (mon.0) 177 : audit [DBG] from='client.? 172.21.15.88:0/928827573' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:13:42.586 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:13:42.586 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:13:43.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:43 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:13:41.653235+0000 mgr.smithi052.ghqrtk (mgr.14162) 120 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:44.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:44.482+0000 7f85b95cd700 1 -- 172.21.15.88:0/3123603292 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f85b40f47c0 msgr2=0x7f85b40f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:44.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:44.482+0000 7f85b95cd700 1 --2- 172.21.15.88:0/3123603292 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f85b40f47c0 0x7f85b40f4be0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f85a4009fb0 tx=0x7f85a4009b30).stop 2024-02-19T22:13:44.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:44.482+0000 7f85b95cd700 1 -- 172.21.15.88:0/3123603292 shutdown_connections 2024-02-19T22:13:44.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:44.482+0000 7f85b95cd700 1 --2- 172.21.15.88:0/3123603292 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f85b40f47c0 0x7f85b40f4be0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:44.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:44.482+0000 7f85b95cd700 1 -- 172.21.15.88:0/3123603292 >> 172.21.15.88:0/3123603292 conn(0x7f85b40f02e0 msgr2=0x7f85b40f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:44.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:44.482+0000 7f85b95cd700 1 -- 172.21.15.88:0/3123603292 shutdown_connections 2024-02-19T22:13:44.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:44.482+0000 7f85b95cd700 1 -- 172.21.15.88:0/3123603292 wait complete. 2024-02-19T22:13:44.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:44.483+0000 7f85b95cd700 1 Processor -- start 2024-02-19T22:13:44.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:44.483+0000 7f85b95cd700 1 -- start start 2024-02-19T22:13:44.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:44.483+0000 7f85b95cd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f85b40f47c0 0x7f85b41067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:44.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:44.483+0000 7f85b95cd700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b4106cc0 con 0x7f85b40f47c0 2024-02-19T22:13:44.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:44.483+0000 7f85b2ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f85b40f47c0 0x7f85b41067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:44.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:44.483+0000 7f85b2ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f85b40f47c0 0x7f85b41067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:44890/0 (socket says 172.21.15.88:44890) 2024-02-19T22:13:44.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:44.483+0000 7f85b2ffd700 1 -- 172.21.15.88:0/751438486 learned_addr learned my addr 172.21.15.88:0/751438486 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:13:44.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:44.484+0000 7f85b2ffd700 1 -- 172.21.15.88:0/751438486 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85a4005040 con 0x7f85b40f47c0 2024-02-19T22:13:44.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:44.484+0000 7f85b2ffd700 1 --2- 172.21.15.88:0/751438486 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f85b40f47c0 0x7f85b41067d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f85a4005010 tx=0x7f85a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:44.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:44.484+0000 7f859b7fe700 1 -- 172.21.15.88:0/751438486 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85a4009320 con 0x7f85b40f47c0 2024-02-19T22:13:44.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:44.484+0000 7f85b95cd700 1 -- 172.21.15.88:0/751438486 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85b4106ec0 con 0x7f85b40f47c0 2024-02-19T22:13:44.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:44.484+0000 7f859b7fe700 1 -- 172.21.15.88:0/751438486 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f85a4013380 con 0x7f85b40f47c0 2024-02-19T22:13:44.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:44.484+0000 7f859b7fe700 1 -- 172.21.15.88:0/751438486 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85a4014e50 con 0x7f85b40f47c0 2024-02-19T22:13:44.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:44.484+0000 7f85b95cd700 1 -- 172.21.15.88:0/751438486 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85b4107350 con 0x7f85b40f47c0 2024-02-19T22:13:44.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:44.485+0000 7f859b7fe700 1 -- 172.21.15.88:0/751438486 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f85a40095f0 con 0x7f85b40f47c0 2024-02-19T22:13:44.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:44.485+0000 7f85b95cd700 1 -- 172.21.15.88:0/751438486 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85b4045ad0 con 0x7f85b40f47c0 2024-02-19T22:13:44.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:44.485+0000 7f859b7fe700 1 --2- 172.21.15.88:0/751438486 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f859c02ff10 0x7f859c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:44.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:44.485+0000 7f859b7fe700 1 -- 172.21.15.88:0/751438486 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f85a4040f80 con 0x7f85b40f47c0 2024-02-19T22:13:44.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:44.485+0000 7f85b27fc700 1 --2- 172.21.15.88:0/751438486 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f859c02ff10 0x7f859c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:44.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:44.486+0000 7f85b27fc700 1 --2- 172.21.15.88:0/751438486 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f859c02ff10 0x7f859c0323d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f85a80096a0 tx=0x7f85a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:44.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:44.490+0000 7f859b7fe700 1 -- 172.21.15.88:0/751438486 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f85a4014420 con 0x7f85b40f47c0 2024-02-19T22:13:44.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:44 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:13:43.748129+0000 mon.smithi052 (mon.0) 178 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi052.ghqrtk/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:13:44.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:44 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:13:43.751748+0000 mon.smithi052 (mon.0) 179 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi052.ghqrtk/trash_purge_schedule"}]: dispatch 2024-02-19T22:13:44.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:44.694+0000 7f85b95cd700 1 -- 172.21.15.88:0/751438486 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f85b4108200 con 0x7f85b40f47c0 2024-02-19T22:13:44.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:44.695+0000 7f859b7fe700 1 -- 172.21.15.88:0/751438486 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f85a4010230 con 0x7f85b40f47c0 2024-02-19T22:13:44.697 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:13:44.697 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:13:44.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:44.697+0000 7f85b95cd700 1 -- 172.21.15.88:0/751438486 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f859c02ff10 msgr2=0x7f859c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:44.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:44.697+0000 7f85b95cd700 1 --2- 172.21.15.88:0/751438486 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f859c02ff10 0x7f859c0323d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f85a80096a0 tx=0x7f85a8006b40).stop 2024-02-19T22:13:44.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:44.697+0000 7f85b95cd700 1 -- 172.21.15.88:0/751438486 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f85b40f47c0 msgr2=0x7f85b41067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:44.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:44.697+0000 7f85b95cd700 1 --2- 172.21.15.88:0/751438486 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f85b40f47c0 0x7f85b41067d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f85a4005010 tx=0x7f85a400b040).stop 2024-02-19T22:13:44.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:44.697+0000 7f85b95cd700 1 -- 172.21.15.88:0/751438486 shutdown_connections 2024-02-19T22:13:44.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:44.697+0000 7f85b95cd700 1 --2- 172.21.15.88:0/751438486 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f859c02ff10 0x7f859c0323d0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:44.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:44.697+0000 7f85b95cd700 1 --2- 172.21.15.88:0/751438486 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f85b40f47c0 0x7f85b41067d0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:44.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:44.697+0000 7f85b95cd700 1 -- 172.21.15.88:0/751438486 >> 172.21.15.88:0/751438486 conn(0x7f85b40f02e0 msgr2=0x7f85b40f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:44.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:44.697+0000 7f85b95cd700 1 -- 172.21.15.88:0/751438486 shutdown_connections 2024-02-19T22:13:44.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:44.697+0000 7f85b95cd700 1 -- 172.21.15.88:0/751438486 wait complete. 2024-02-19T22:13:44.700 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:13:45.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:45 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:13:43.653567+0000 mgr.smithi052.ghqrtk (mgr.14162) 121 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:45.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:45 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:13:44.696530+0000 mon.smithi052 (mon.0) 180 : audit [DBG] from='client.? 172.21.15.88:0/751438486' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:13:46.323 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:13:46.323 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:13:47.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:47 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:13:45.653870+0000 mgr.smithi052.ghqrtk (mgr.14162) 122 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:48.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:48.063+0000 7f8c6c6d3700 1 -- 172.21.15.88:0/4078588608 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8c64060280 msgr2=0x7f8c640f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:48.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:48.063+0000 7f8c6c6d3700 1 --2- 172.21.15.88:0/4078588608 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8c64060280 0x7f8c640f90f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f8c54009fb0 tx=0x7f8c54009b30).stop 2024-02-19T22:13:48.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:48.063+0000 7f8c6c6d3700 1 -- 172.21.15.88:0/4078588608 shutdown_connections 2024-02-19T22:13:48.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:48.063+0000 7f8c6c6d3700 1 --2- 172.21.15.88:0/4078588608 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8c64060280 0x7f8c640f90f0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:48.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:48.063+0000 7f8c6c6d3700 1 -- 172.21.15.88:0/4078588608 >> 172.21.15.88:0/4078588608 conn(0x7f8c640f0b70 msgr2=0x7f8c640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:48.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:48.063+0000 7f8c6c6d3700 1 -- 172.21.15.88:0/4078588608 shutdown_connections 2024-02-19T22:13:48.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:48.063+0000 7f8c6c6d3700 1 -- 172.21.15.88:0/4078588608 wait complete. 2024-02-19T22:13:48.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:48.064+0000 7f8c6c6d3700 1 Processor -- start 2024-02-19T22:13:48.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:48.064+0000 7f8c6c6d3700 1 -- start start 2024-02-19T22:13:48.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:48.064+0000 7f8c6c6d3700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8c64060280 0x7f8c64106760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:48.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:48.064+0000 7f8c6c6d3700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c64106c50 con 0x7f8c64060280 2024-02-19T22:13:48.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:48.064+0000 7f8c6a46f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8c64060280 0x7f8c64106760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:48.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:48.064+0000 7f8c6a46f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8c64060280 0x7f8c64106760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:44910/0 (socket says 172.21.15.88:44910) 2024-02-19T22:13:48.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:48.064+0000 7f8c6a46f700 1 -- 172.21.15.88:0/2108863547 learned_addr learned my addr 172.21.15.88:0/2108863547 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:13:48.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:48.064+0000 7f8c6a46f700 1 -- 172.21.15.88:0/2108863547 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c54005040 con 0x7f8c64060280 2024-02-19T22:13:48.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:48.064+0000 7f8c6a46f700 1 --2- 172.21.15.88:0/2108863547 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8c64060280 0x7f8c64106760 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f8c540089f0 tx=0x7f8c5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:48.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:48.065+0000 7f8c5affd700 1 -- 172.21.15.88:0/2108863547 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c54009320 con 0x7f8c64060280 2024-02-19T22:13:48.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:48.065+0000 7f8c6c6d3700 1 -- 172.21.15.88:0/2108863547 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c64106e50 con 0x7f8c64060280 2024-02-19T22:13:48.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:48.065+0000 7f8c5affd700 1 -- 172.21.15.88:0/2108863547 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8c54013380 con 0x7f8c64060280 2024-02-19T22:13:48.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:48.065+0000 7f8c5affd700 1 -- 172.21.15.88:0/2108863547 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c54014e20 con 0x7f8c64060280 2024-02-19T22:13:48.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:48.065+0000 7f8c6c6d3700 1 -- 172.21.15.88:0/2108863547 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c641072e0 con 0x7f8c64060280 2024-02-19T22:13:48.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:48.066+0000 7f8c5affd700 1 -- 172.21.15.88:0/2108863547 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8c540095f0 con 0x7f8c64060280 2024-02-19T22:13:48.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:48.066+0000 7f8c6c6d3700 1 -- 172.21.15.88:0/2108863547 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c64045ad0 con 0x7f8c64060280 2024-02-19T22:13:48.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:48.066+0000 7f8c5affd700 1 --2- 172.21.15.88:0/2108863547 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f8c5002ff10 0x7f8c500323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:48.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:48.066+0000 7f8c5affd700 1 -- 172.21.15.88:0/2108863547 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8c54040f10 con 0x7f8c64060280 2024-02-19T22:13:48.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:48.066+0000 7f8c69c6e700 1 --2- 172.21.15.88:0/2108863547 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f8c5002ff10 0x7f8c500323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:48.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:48.066+0000 7f8c69c6e700 1 --2- 172.21.15.88:0/2108863547 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f8c5002ff10 0x7f8c500323d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f8c600096a0 tx=0x7f8c60006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:48.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:48.070+0000 7f8c5affd700 1 -- 172.21.15.88:0/2108863547 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8c54014420 con 0x7f8c64060280 2024-02-19T22:13:48.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:48.275+0000 7f8c6c6d3700 1 -- 172.21.15.88:0/2108863547 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8c64108190 con 0x7f8c64060280 2024-02-19T22:13:48.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:48.277+0000 7f8c5affd700 1 -- 172.21.15.88:0/2108863547 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8c5401d980 con 0x7f8c64060280 2024-02-19T22:13:48.278 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:13:48.278 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:13:48.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:48.277+0000 7f8c6c6d3700 1 -- 172.21.15.88:0/2108863547 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f8c5002ff10 msgr2=0x7f8c500323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:48.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:48.277+0000 7f8c6c6d3700 1 --2- 172.21.15.88:0/2108863547 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f8c5002ff10 0x7f8c500323d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f8c600096a0 tx=0x7f8c60006b40).stop 2024-02-19T22:13:48.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:48.277+0000 7f8c6c6d3700 1 -- 172.21.15.88:0/2108863547 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8c64060280 msgr2=0x7f8c64106760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:48.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:48.277+0000 7f8c6c6d3700 1 --2- 172.21.15.88:0/2108863547 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8c64060280 0x7f8c64106760 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f8c540089f0 tx=0x7f8c5400b040).stop 2024-02-19T22:13:48.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:48.277+0000 7f8c6c6d3700 1 -- 172.21.15.88:0/2108863547 shutdown_connections 2024-02-19T22:13:48.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:48.277+0000 7f8c6c6d3700 1 --2- 172.21.15.88:0/2108863547 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f8c5002ff10 0x7f8c500323d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:48.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:48.277+0000 7f8c6c6d3700 1 --2- 172.21.15.88:0/2108863547 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8c64060280 0x7f8c64106760 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:48.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:48.278+0000 7f8c6c6d3700 1 -- 172.21.15.88:0/2108863547 >> 172.21.15.88:0/2108863547 conn(0x7f8c640f0b70 msgr2=0x7f8c640f15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:48.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:48.278+0000 7f8c6c6d3700 1 -- 172.21.15.88:0/2108863547 shutdown_connections 2024-02-19T22:13:48.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:48.278+0000 7f8c6c6d3700 1 -- 172.21.15.88:0/2108863547 wait complete. 2024-02-19T22:13:48.281 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:13:49.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:49 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:13:47.654182+0000 mgr.smithi052.ghqrtk (mgr.14162) 123 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:49.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:49 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:13:48.277401+0000 mon.smithi052 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.88:0/2108863547' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:13:50.004 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:13:50.004 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:13:51.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:51 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:13:49.654515+0000 mgr.smithi052.ghqrtk (mgr.14162) 124 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:51.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:51.750+0000 7fc0f3b78700 1 -- 172.21.15.88:0/2428580299 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc0ec0f6510 msgr2=0x7fc0ec0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:51.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:51.750+0000 7fc0f3b78700 1 --2- 172.21.15.88:0/2428580299 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc0ec0f6510 0x7fc0ec0f6930 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7fc0dc009fb0 tx=0x7fc0dc009b30).stop 2024-02-19T22:13:51.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:51.750+0000 7fc0f3b78700 1 -- 172.21.15.88:0/2428580299 shutdown_connections 2024-02-19T22:13:51.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:51.750+0000 7fc0f3b78700 1 --2- 172.21.15.88:0/2428580299 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc0ec0f6510 0x7fc0ec0f6930 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:51.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:51.750+0000 7fc0f3b78700 1 -- 172.21.15.88:0/2428580299 >> 172.21.15.88:0/2428580299 conn(0x7fc0ec0f1c70 msgr2=0x7fc0ec0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:51.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:51.750+0000 7fc0f3b78700 1 -- 172.21.15.88:0/2428580299 shutdown_connections 2024-02-19T22:13:51.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:51.750+0000 7fc0f3b78700 1 -- 172.21.15.88:0/2428580299 wait complete. 2024-02-19T22:13:51.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:51.751+0000 7fc0f3b78700 1 Processor -- start 2024-02-19T22:13:51.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:51.751+0000 7fc0f3b78700 1 -- start start 2024-02-19T22:13:51.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:51.751+0000 7fc0f3b78700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc0ec0f6510 0x7fc0ec1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:51.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:51.751+0000 7fc0f3b78700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0ec106cd0 con 0x7fc0ec0f6510 2024-02-19T22:13:51.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:51.751+0000 7fc0f1914700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc0ec0f6510 0x7fc0ec1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:51.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:51.751+0000 7fc0f1914700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc0ec0f6510 0x7fc0ec1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:44928/0 (socket says 172.21.15.88:44928) 2024-02-19T22:13:51.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:51.751+0000 7fc0f1914700 1 -- 172.21.15.88:0/1602778535 learned_addr learned my addr 172.21.15.88:0/1602778535 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:13:51.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:51.751+0000 7fc0f1914700 1 -- 172.21.15.88:0/1602778535 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0dc005040 con 0x7fc0ec0f6510 2024-02-19T22:13:51.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:51.751+0000 7fc0f1914700 1 --2- 172.21.15.88:0/1602778535 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc0ec0f6510 0x7fc0ec1067e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7fc0dc005010 tx=0x7fc0dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:51.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:51.751+0000 7fc0e27fc700 1 -- 172.21.15.88:0/1602778535 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0dc009320 con 0x7fc0ec0f6510 2024-02-19T22:13:51.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:51.751+0000 7fc0f3b78700 1 -- 172.21.15.88:0/1602778535 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0ec106ed0 con 0x7fc0ec0f6510 2024-02-19T22:13:51.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:51.752+0000 7fc0e27fc700 1 -- 172.21.15.88:0/1602778535 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc0dc013380 con 0x7fc0ec0f6510 2024-02-19T22:13:51.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:51.752+0000 7fc0e27fc700 1 -- 172.21.15.88:0/1602778535 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0dc014e50 con 0x7fc0ec0f6510 2024-02-19T22:13:51.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:51.752+0000 7fc0f3b78700 1 -- 172.21.15.88:0/1602778535 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0ec107360 con 0x7fc0ec0f6510 2024-02-19T22:13:51.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:51.752+0000 7fc0e27fc700 1 -- 172.21.15.88:0/1602778535 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc0dc0263a0 con 0x7fc0ec0f6510 2024-02-19T22:13:51.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:51.752+0000 7fc0f3b78700 1 -- 172.21.15.88:0/1602778535 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0ec045ad0 con 0x7fc0ec0f6510 2024-02-19T22:13:51.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:51.752+0000 7fc0e27fc700 1 --2- 172.21.15.88:0/1602778535 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fc0d802ff10 0x7fc0d80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:51.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:51.752+0000 7fc0e27fc700 1 -- 172.21.15.88:0/1602778535 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc0dc040fd0 con 0x7fc0ec0f6510 2024-02-19T22:13:51.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:51.753+0000 7fc0f1113700 1 --2- 172.21.15.88:0/1602778535 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fc0d802ff10 0x7fc0d80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:51.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:51.753+0000 7fc0f1113700 1 --2- 172.21.15.88:0/1602778535 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fc0d802ff10 0x7fc0d80323d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fc0e80096a0 tx=0x7fc0e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:51.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:51.758+0000 7fc0e27fc700 1 -- 172.21.15.88:0/1602778535 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc0dc0095f0 con 0x7fc0ec0f6510 2024-02-19T22:13:51.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:51.962+0000 7fc0f3b78700 1 -- 172.21.15.88:0/1602778535 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc0ec108210 con 0x7fc0ec0f6510 2024-02-19T22:13:51.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:51.964+0000 7fc0e27fc700 1 -- 172.21.15.88:0/1602778535 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc0dc01d980 con 0x7fc0ec0f6510 2024-02-19T22:13:51.965 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:13:51.965 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:13:51.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:51.964+0000 7fc0f3b78700 1 -- 172.21.15.88:0/1602778535 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fc0d802ff10 msgr2=0x7fc0d80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:51.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:51.964+0000 7fc0f3b78700 1 --2- 172.21.15.88:0/1602778535 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fc0d802ff10 0x7fc0d80323d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fc0e80096a0 tx=0x7fc0e8006b40).stop 2024-02-19T22:13:51.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:51.964+0000 7fc0f3b78700 1 -- 172.21.15.88:0/1602778535 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc0ec0f6510 msgr2=0x7fc0ec1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:51.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:51.964+0000 7fc0f3b78700 1 --2- 172.21.15.88:0/1602778535 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc0ec0f6510 0x7fc0ec1067e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7fc0dc005010 tx=0x7fc0dc00b040).stop 2024-02-19T22:13:51.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:51.964+0000 7fc0f3b78700 1 -- 172.21.15.88:0/1602778535 shutdown_connections 2024-02-19T22:13:51.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:51.964+0000 7fc0f3b78700 1 --2- 172.21.15.88:0/1602778535 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fc0d802ff10 0x7fc0d80323d0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:51.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:51.964+0000 7fc0f3b78700 1 --2- 172.21.15.88:0/1602778535 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc0ec0f6510 0x7fc0ec1067e0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:51.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:51.964+0000 7fc0f3b78700 1 -- 172.21.15.88:0/1602778535 >> 172.21.15.88:0/1602778535 conn(0x7fc0ec0f1c70 msgr2=0x7fc0ec0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:51.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:51.965+0000 7fc0f3b78700 1 -- 172.21.15.88:0/1602778535 shutdown_connections 2024-02-19T22:13:51.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:51.965+0000 7fc0f3b78700 1 -- 172.21.15.88:0/1602778535 wait complete. 2024-02-19T22:13:51.967 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:13:52.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:52 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:13:51.964404+0000 mon.smithi052 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.88:0/1602778535' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:13:53.586 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:13:53.587 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:13:53.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:53 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:13:51.654877+0000 mgr.smithi052.ghqrtk (mgr.14162) 125 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:55.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:55.427+0000 7fa2b27c2700 1 -- 172.21.15.88:0/745774297 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa2ac0f6510 msgr2=0x7fa2ac0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:55.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:55.427+0000 7fa2b27c2700 1 --2- 172.21.15.88:0/745774297 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa2ac0f6510 0x7fa2ac0f6930 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7fa29c009fb0 tx=0x7fa29c009b30).stop 2024-02-19T22:13:55.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:55.427+0000 7fa2b27c2700 1 -- 172.21.15.88:0/745774297 shutdown_connections 2024-02-19T22:13:55.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:55.427+0000 7fa2b27c2700 1 --2- 172.21.15.88:0/745774297 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa2ac0f6510 0x7fa2ac0f6930 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:55.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:55.427+0000 7fa2b27c2700 1 -- 172.21.15.88:0/745774297 >> 172.21.15.88:0/745774297 conn(0x7fa2ac0f1c70 msgr2=0x7fa2ac0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:55.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:55.427+0000 7fa2b27c2700 1 -- 172.21.15.88:0/745774297 shutdown_connections 2024-02-19T22:13:55.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:55.427+0000 7fa2b27c2700 1 -- 172.21.15.88:0/745774297 wait complete. 2024-02-19T22:13:55.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:55.428+0000 7fa2b27c2700 1 Processor -- start 2024-02-19T22:13:55.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:55.428+0000 7fa2b27c2700 1 -- start start 2024-02-19T22:13:55.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:55.428+0000 7fa2b27c2700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa2ac0f6510 0x7fa2ac06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:55.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:55.428+0000 7fa2b27c2700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ac069e20 con 0x7fa2ac0f6510 2024-02-19T22:13:55.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:55.428+0000 7fa2abfff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa2ac0f6510 0x7fa2ac06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:55.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:55.428+0000 7fa2abfff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa2ac0f6510 0x7fa2ac06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:51056/0 (socket says 172.21.15.88:51056) 2024-02-19T22:13:55.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:55.428+0000 7fa2abfff700 1 -- 172.21.15.88:0/3608249587 learned_addr learned my addr 172.21.15.88:0/3608249587 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:13:55.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:55.428+0000 7fa2abfff700 1 -- 172.21.15.88:0/3608249587 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa29c005040 con 0x7fa2ac0f6510 2024-02-19T22:13:55.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:55.428+0000 7fa2abfff700 1 --2- 172.21.15.88:0/3608249587 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa2ac0f6510 0x7fa2ac06b9a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fa29c005010 tx=0x7fa29c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:55.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:55.429+0000 7fa2a8ff9700 1 -- 172.21.15.88:0/3608249587 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa29c009320 con 0x7fa2ac0f6510 2024-02-19T22:13:55.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:55.429+0000 7fa2b27c2700 1 -- 172.21.15.88:0/3608249587 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2ac069fc0 con 0x7fa2ac0f6510 2024-02-19T22:13:55.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:55.429+0000 7fa2a8ff9700 1 -- 172.21.15.88:0/3608249587 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa29c013380 con 0x7fa2ac0f6510 2024-02-19T22:13:55.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:55.429+0000 7fa2a8ff9700 1 -- 172.21.15.88:0/3608249587 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa29c014e50 con 0x7fa2ac0f6510 2024-02-19T22:13:55.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:55.429+0000 7fa2b27c2700 1 -- 172.21.15.88:0/3608249587 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2ac06a450 con 0x7fa2ac0f6510 2024-02-19T22:13:55.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:55.430+0000 7fa2a8ff9700 1 -- 172.21.15.88:0/3608249587 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa29c0263a0 con 0x7fa2ac0f6510 2024-02-19T22:13:55.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:55.430+0000 7fa2b27c2700 1 -- 172.21.15.88:0/3608249587 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2ac059100 con 0x7fa2ac0f6510 2024-02-19T22:13:55.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:55.430+0000 7fa2a8ff9700 1 --2- 172.21.15.88:0/3608249587 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa29802ff10 0x7fa2980323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:55.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:55.430+0000 7fa2a8ff9700 1 -- 172.21.15.88:0/3608249587 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa29c041140 con 0x7fa2ac0f6510 2024-02-19T22:13:55.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:55.430+0000 7fa2ab7fe700 1 --2- 172.21.15.88:0/3608249587 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa29802ff10 0x7fa2980323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:55.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:55.431+0000 7fa2ab7fe700 1 --2- 172.21.15.88:0/3608249587 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa29802ff10 0x7fa2980323d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fa2940096a0 tx=0x7fa294006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:55.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:55.435+0000 7fa2a8ff9700 1 -- 172.21.15.88:0/3608249587 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa29c01cb10 con 0x7fa2ac0f6510 2024-02-19T22:13:55.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:55 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:13:53.655174+0000 mgr.smithi052.ghqrtk (mgr.14162) 126 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:55.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:55.638+0000 7fa2b27c2700 1 -- 172.21.15.88:0/3608249587 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa2ac0fc0a0 con 0x7fa2ac0f6510 2024-02-19T22:13:55.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:55.639+0000 7fa2a8ff9700 1 -- 172.21.15.88:0/3608249587 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa29c01cb10 con 0x7fa2ac0f6510 2024-02-19T22:13:55.639 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:13:55.640 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:13:55.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:55.640+0000 7fa2b27c2700 1 -- 172.21.15.88:0/3608249587 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa29802ff10 msgr2=0x7fa2980323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:55.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:55.640+0000 7fa2b27c2700 1 --2- 172.21.15.88:0/3608249587 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa29802ff10 0x7fa2980323d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fa2940096a0 tx=0x7fa294006b40).stop 2024-02-19T22:13:55.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:55.640+0000 7fa2b27c2700 1 -- 172.21.15.88:0/3608249587 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa2ac0f6510 msgr2=0x7fa2ac06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:55.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:55.640+0000 7fa2b27c2700 1 --2- 172.21.15.88:0/3608249587 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa2ac0f6510 0x7fa2ac06b9a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fa29c005010 tx=0x7fa29c00b040).stop 2024-02-19T22:13:55.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:55.640+0000 7fa2b27c2700 1 -- 172.21.15.88:0/3608249587 shutdown_connections 2024-02-19T22:13:55.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:55.640+0000 7fa2b27c2700 1 --2- 172.21.15.88:0/3608249587 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa29802ff10 0x7fa2980323d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:55.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:55.640+0000 7fa2b27c2700 1 --2- 172.21.15.88:0/3608249587 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa2ac0f6510 0x7fa2ac06b9a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:55.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:55.640+0000 7fa2b27c2700 1 -- 172.21.15.88:0/3608249587 >> 172.21.15.88:0/3608249587 conn(0x7fa2ac0f1c70 msgr2=0x7fa2ac0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:55.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:55.640+0000 7fa2b27c2700 1 -- 172.21.15.88:0/3608249587 shutdown_connections 2024-02-19T22:13:55.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:55.640+0000 7fa2b27c2700 1 -- 172.21.15.88:0/3608249587 wait complete. 2024-02-19T22:13:55.642 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:13:56.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:13:55.639248+0000 mon.smithi052 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.88:0/3608249587' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:13:57.268 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:13:57.268 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:13:57.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:57 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:13:55.655454+0000 mgr.smithi052.ghqrtk (mgr.14162) 127 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:59.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:59.080+0000 7fdde1277700 1 -- 172.21.15.88:0/2274382556 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdddc0f6510 msgr2=0x7fdddc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:59.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:59.080+0000 7fdde1277700 1 --2- 172.21.15.88:0/2274382556 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdddc0f6510 0x7fdddc0f6930 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7fddd0009fb0 tx=0x7fddd0009b30).stop 2024-02-19T22:13:59.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:59.080+0000 7fdde1277700 1 -- 172.21.15.88:0/2274382556 shutdown_connections 2024-02-19T22:13:59.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:59.080+0000 7fdde1277700 1 --2- 172.21.15.88:0/2274382556 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdddc0f6510 0x7fdddc0f6930 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:59.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:59.080+0000 7fdde1277700 1 -- 172.21.15.88:0/2274382556 >> 172.21.15.88:0/2274382556 conn(0x7fdddc0f1c70 msgr2=0x7fdddc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:59.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:59.080+0000 7fdde1277700 1 -- 172.21.15.88:0/2274382556 shutdown_connections 2024-02-19T22:13:59.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:59.080+0000 7fdde1277700 1 -- 172.21.15.88:0/2274382556 wait complete. 2024-02-19T22:13:59.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:59.080+0000 7fdde1277700 1 Processor -- start 2024-02-19T22:13:59.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:59.080+0000 7fdde1277700 1 -- start start 2024-02-19T22:13:59.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:59.081+0000 7fdde1277700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdddc0f6510 0x7fdddc06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:59.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:59.081+0000 7fdde1277700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdddc069e20 con 0x7fdddc0f6510 2024-02-19T22:13:59.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:59.081+0000 7fdddaffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdddc0f6510 0x7fdddc06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:59.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:59.081+0000 7fdddaffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdddc0f6510 0x7fdddc06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:51072/0 (socket says 172.21.15.88:51072) 2024-02-19T22:13:59.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:59.081+0000 7fdddaffd700 1 -- 172.21.15.88:0/1619905409 learned_addr learned my addr 172.21.15.88:0/1619905409 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:13:59.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:59.081+0000 7fdddaffd700 1 -- 172.21.15.88:0/1619905409 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd0005040 con 0x7fdddc0f6510 2024-02-19T22:13:59.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:59.081+0000 7fdddaffd700 1 --2- 172.21.15.88:0/1619905409 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdddc0f6510 0x7fdddc06b9a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fddd00089f0 tx=0x7fddd000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:59.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:59.081+0000 7fddc37fe700 1 -- 172.21.15.88:0/1619905409 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fddd0009320 con 0x7fdddc0f6510 2024-02-19T22:13:59.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:59.081+0000 7fddc37fe700 1 -- 172.21.15.88:0/1619905409 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fddd0013380 con 0x7fdddc0f6510 2024-02-19T22:13:59.084 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:59.081+0000 7fdde1277700 1 -- 172.21.15.88:0/1619905409 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdddc069fc0 con 0x7fdddc0f6510 2024-02-19T22:13:59.084 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:59.081+0000 7fddc37fe700 1 -- 172.21.15.88:0/1619905409 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fddd001ed80 con 0x7fdddc0f6510 2024-02-19T22:13:59.084 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:59.081+0000 7fdde1277700 1 -- 172.21.15.88:0/1619905409 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdddc06a450 con 0x7fdddc0f6510 2024-02-19T22:13:59.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:59.082+0000 7fddc37fe700 1 -- 172.21.15.88:0/1619905409 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fddd001e420 con 0x7fdddc0f6510 2024-02-19T22:13:59.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:59.082+0000 7fdde1277700 1 -- 172.21.15.88:0/1619905409 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdddc059100 con 0x7fdddc0f6510 2024-02-19T22:13:59.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:59.082+0000 7fddc37fe700 1 --2- 172.21.15.88:0/1619905409 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fddc402ff60 0x7fddc4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:59.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:59.082+0000 7fddc37fe700 1 -- 172.21.15.88:0/1619905409 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fddd0040f30 con 0x7fdddc0f6510 2024-02-19T22:13:59.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:59.083+0000 7fddda7fc700 1 --2- 172.21.15.88:0/1619905409 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fddc402ff60 0x7fddc4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:59.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:59.083+0000 7fddda7fc700 1 --2- 172.21.15.88:0/1619905409 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fddc402ff60 0x7fddc4032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fddcc0096a0 tx=0x7fddcc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:59.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:59.087+0000 7fddc37fe700 1 -- 172.21.15.88:0/1619905409 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fddd000e2a0 con 0x7fdddc0f6510 2024-02-19T22:13:59.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:59.289+0000 7fdde1277700 1 -- 172.21.15.88:0/1619905409 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdddc0fc0a0 con 0x7fdddc0f6510 2024-02-19T22:13:59.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:59.290+0000 7fddc37fe700 1 -- 172.21.15.88:0/1619905409 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fddd000e2a0 con 0x7fdddc0f6510 2024-02-19T22:13:59.291 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:13:59.291 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:13:59.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:59.292+0000 7fdde1277700 1 -- 172.21.15.88:0/1619905409 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fddc402ff60 msgr2=0x7fddc4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:59.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:59.292+0000 7fdde1277700 1 --2- 172.21.15.88:0/1619905409 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fddc402ff60 0x7fddc4032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fddcc0096a0 tx=0x7fddcc006b40).stop 2024-02-19T22:13:59.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:59.292+0000 7fdde1277700 1 -- 172.21.15.88:0/1619905409 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdddc0f6510 msgr2=0x7fdddc06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:59.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:59.292+0000 7fdde1277700 1 --2- 172.21.15.88:0/1619905409 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdddc0f6510 0x7fdddc06b9a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fddd00089f0 tx=0x7fddd000b040).stop 2024-02-19T22:13:59.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:59.292+0000 7fdde1277700 1 -- 172.21.15.88:0/1619905409 shutdown_connections 2024-02-19T22:13:59.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:59.292+0000 7fdde1277700 1 --2- 172.21.15.88:0/1619905409 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fddc402ff60 0x7fddc4032420 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:59.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:59.292+0000 7fdde1277700 1 --2- 172.21.15.88:0/1619905409 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdddc0f6510 0x7fdddc06b9a0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:59.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:59.292+0000 7fdde1277700 1 -- 172.21.15.88:0/1619905409 >> 172.21.15.88:0/1619905409 conn(0x7fdddc0f1c70 msgr2=0x7fdddc0f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:59.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:59.292+0000 7fdde1277700 1 -- 172.21.15.88:0/1619905409 shutdown_connections 2024-02-19T22:13:59.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:13:59.292+0000 7fdde1277700 1 -- 172.21.15.88:0/1619905409 wait complete. 2024-02-19T22:13:59.294 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:13:59.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:13:59 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:13:57.655754+0000 mgr.smithi052.ghqrtk (mgr.14162) 128 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:00.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:00 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:13:59.290544+0000 mon.smithi052 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.88:0/1619905409' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:00.874 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:00.874 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:01.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:01 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:13:59.656049+0000 mgr.smithi052.ghqrtk (mgr.14162) 129 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:02.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:02.637+0000 7fd87a7a4700 1 -- 172.21.15.88:0/3611818868 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8740f5410 msgr2=0x7fd8740f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:02.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:02.637+0000 7fd87a7a4700 1 --2- 172.21.15.88:0/3611818868 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8740f5410 0x7fd8740f5830 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7fd864009fb0 tx=0x7fd864009b30).stop 2024-02-19T22:14:02.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:02.637+0000 7fd87a7a4700 1 -- 172.21.15.88:0/3611818868 shutdown_connections 2024-02-19T22:14:02.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:02.637+0000 7fd87a7a4700 1 --2- 172.21.15.88:0/3611818868 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8740f5410 0x7fd8740f5830 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:02.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:02.637+0000 7fd87a7a4700 1 -- 172.21.15.88:0/3611818868 >> 172.21.15.88:0/3611818868 conn(0x7fd8740f0b70 msgr2=0x7fd8740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:02.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:02.637+0000 7fd87a7a4700 1 -- 172.21.15.88:0/3611818868 shutdown_connections 2024-02-19T22:14:02.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:02.637+0000 7fd87a7a4700 1 -- 172.21.15.88:0/3611818868 wait complete. 2024-02-19T22:14:02.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:02.638+0000 7fd87a7a4700 1 Processor -- start 2024-02-19T22:14:02.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:02.638+0000 7fd87a7a4700 1 -- start start 2024-02-19T22:14:02.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:02.638+0000 7fd87a7a4700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8740f5410 0x7fd8741094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:02.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:02.638+0000 7fd87a7a4700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd874060240 con 0x7fd8740f5410 2024-02-19T22:14:02.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:02.638+0000 7fd873fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8740f5410 0x7fd8741094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:02.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:02.638+0000 7fd873fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8740f5410 0x7fd8741094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:51088/0 (socket says 172.21.15.88:51088) 2024-02-19T22:14:02.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:02.638+0000 7fd873fff700 1 -- 172.21.15.88:0/902796998 learned_addr learned my addr 172.21.15.88:0/902796998 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:14:02.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:02.639+0000 7fd873fff700 1 -- 172.21.15.88:0/902796998 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd864005040 con 0x7fd8740f5410 2024-02-19T22:14:02.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:02.639+0000 7fd873fff700 1 --2- 172.21.15.88:0/902796998 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8740f5410 0x7fd8741094e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fd864005010 tx=0x7fd86400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:02.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:02.639+0000 7fd870ff9700 1 -- 172.21.15.88:0/902796998 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd864009320 con 0x7fd8740f5410 2024-02-19T22:14:02.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:02.639+0000 7fd87a7a4700 1 -- 172.21.15.88:0/902796998 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd874060440 con 0x7fd8740f5410 2024-02-19T22:14:02.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:02.639+0000 7fd870ff9700 1 -- 172.21.15.88:0/902796998 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd864013380 con 0x7fd8740f5410 2024-02-19T22:14:02.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:02.639+0000 7fd870ff9700 1 -- 172.21.15.88:0/902796998 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd864014e50 con 0x7fd8740f5410 2024-02-19T22:14:02.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:02.639+0000 7fd87a7a4700 1 -- 172.21.15.88:0/902796998 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd874109b90 con 0x7fd8740f5410 2024-02-19T22:14:02.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:02.640+0000 7fd870ff9700 1 -- 172.21.15.88:0/902796998 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd8640263a0 con 0x7fd8740f5410 2024-02-19T22:14:02.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:02.640+0000 7fd87a7a4700 1 -- 172.21.15.88:0/902796998 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd874045ad0 con 0x7fd8740f5410 2024-02-19T22:14:02.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:02.640+0000 7fd870ff9700 1 --2- 172.21.15.88:0/902796998 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd86002ff10 0x7fd8600323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:02.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:02.640+0000 7fd870ff9700 1 -- 172.21.15.88:0/902796998 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd864040ff0 con 0x7fd8740f5410 2024-02-19T22:14:02.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:02.640+0000 7fd8737fe700 1 --2- 172.21.15.88:0/902796998 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd86002ff10 0x7fd8600323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:02.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:02.641+0000 7fd8737fe700 1 --2- 172.21.15.88:0/902796998 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd86002ff10 0x7fd8600323d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fd85c0096a0 tx=0x7fd85c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:02.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:02.645+0000 7fd870ff9700 1 -- 172.21.15.88:0/902796998 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd86406d2b0 con 0x7fd8740f5410 2024-02-19T22:14:02.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:02.847+0000 7fd87a7a4700 1 -- 172.21.15.88:0/902796998 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd87410a3b0 con 0x7fd8740f5410 2024-02-19T22:14:02.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:02.847+0000 7fd870ff9700 1 -- 172.21.15.88:0/902796998 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd86401dde0 con 0x7fd8740f5410 2024-02-19T22:14:02.848 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:14:02.849 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:14:02.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:02.849+0000 7fd87a7a4700 1 -- 172.21.15.88:0/902796998 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd86002ff10 msgr2=0x7fd8600323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:02.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:02.849+0000 7fd87a7a4700 1 --2- 172.21.15.88:0/902796998 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd86002ff10 0x7fd8600323d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fd85c0096a0 tx=0x7fd85c006b40).stop 2024-02-19T22:14:02.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:02.849+0000 7fd87a7a4700 1 -- 172.21.15.88:0/902796998 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8740f5410 msgr2=0x7fd8741094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:02.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:02.849+0000 7fd87a7a4700 1 --2- 172.21.15.88:0/902796998 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8740f5410 0x7fd8741094e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fd864005010 tx=0x7fd86400b040).stop 2024-02-19T22:14:02.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:02.849+0000 7fd87a7a4700 1 -- 172.21.15.88:0/902796998 shutdown_connections 2024-02-19T22:14:02.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:02.849+0000 7fd87a7a4700 1 --2- 172.21.15.88:0/902796998 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd86002ff10 0x7fd8600323d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:02.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:02.849+0000 7fd87a7a4700 1 --2- 172.21.15.88:0/902796998 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8740f5410 0x7fd8741094e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:02.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:02.849+0000 7fd87a7a4700 1 -- 172.21.15.88:0/902796998 >> 172.21.15.88:0/902796998 conn(0x7fd8740f0b70 msgr2=0x7fd8740f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:02.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:02.849+0000 7fd87a7a4700 1 -- 172.21.15.88:0/902796998 shutdown_connections 2024-02-19T22:14:02.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:02.849+0000 7fd87a7a4700 1 -- 172.21.15.88:0/902796998 wait complete. 2024-02-19T22:14:02.851 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:14:03.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:03 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:14:01.656370+0000 mgr.smithi052.ghqrtk (mgr.14162) 130 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:03.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:03 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:14:02.847916+0000 mon.smithi052 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.88:0/902796998' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:04.535 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:04.535 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:05.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:05 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:14:03.656721+0000 mgr.smithi052.ghqrtk (mgr.14162) 131 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:06.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:06.504+0000 7f4455ea6700 1 -- 172.21.15.88:0/1360512752 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44500f5410 msgr2=0x7f44500f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:06.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:06.504+0000 7f4455ea6700 1 --2- 172.21.15.88:0/1360512752 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44500f5410 0x7f44500f5830 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f4444009fb0 tx=0x7f4444009b30).stop 2024-02-19T22:14:06.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:06.504+0000 7f4455ea6700 1 -- 172.21.15.88:0/1360512752 shutdown_connections 2024-02-19T22:14:06.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:06.504+0000 7f4455ea6700 1 --2- 172.21.15.88:0/1360512752 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44500f5410 0x7f44500f5830 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:06.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:06.504+0000 7f4455ea6700 1 -- 172.21.15.88:0/1360512752 >> 172.21.15.88:0/1360512752 conn(0x7f44500f0b70 msgr2=0x7f44500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:06.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:06.504+0000 7f4455ea6700 1 -- 172.21.15.88:0/1360512752 shutdown_connections 2024-02-19T22:14:06.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:06.504+0000 7f4455ea6700 1 -- 172.21.15.88:0/1360512752 wait complete. 2024-02-19T22:14:06.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:06.505+0000 7f4455ea6700 1 Processor -- start 2024-02-19T22:14:06.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:06.505+0000 7f4455ea6700 1 -- start start 2024-02-19T22:14:06.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:06.505+0000 7f4455ea6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44500f5410 0x7f44501089b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:06.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:06.505+0000 7f4455ea6700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4450108ea0 con 0x7f44500f5410 2024-02-19T22:14:06.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:06.505+0000 7f444f7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44500f5410 0x7f44501089b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:06.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:06.505+0000 7f444f7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44500f5410 0x7f44501089b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:38922/0 (socket says 172.21.15.88:38922) 2024-02-19T22:14:06.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:06.505+0000 7f444f7fe700 1 -- 172.21.15.88:0/2531164054 learned_addr learned my addr 172.21.15.88:0/2531164054 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:14:06.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:06.506+0000 7f444f7fe700 1 -- 172.21.15.88:0/2531164054 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4444005040 con 0x7f44500f5410 2024-02-19T22:14:06.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:06.506+0000 7f444f7fe700 1 --2- 172.21.15.88:0/2531164054 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44500f5410 0x7f44501089b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f4444005010 tx=0x7f444400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:06.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:06.506+0000 7f4437fff700 1 -- 172.21.15.88:0/2531164054 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4444009320 con 0x7f44500f5410 2024-02-19T22:14:06.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:06.506+0000 7f4437fff700 1 -- 172.21.15.88:0/2531164054 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4444013380 con 0x7f44500f5410 2024-02-19T22:14:06.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:06.506+0000 7f4455ea6700 1 -- 172.21.15.88:0/2531164054 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44501090a0 con 0x7f44500f5410 2024-02-19T22:14:06.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:06.506+0000 7f4437fff700 1 -- 172.21.15.88:0/2531164054 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4444014e50 con 0x7f44500f5410 2024-02-19T22:14:06.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:06.506+0000 7f4455ea6700 1 -- 172.21.15.88:0/2531164054 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4450109530 con 0x7f44500f5410 2024-02-19T22:14:06.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:06.507+0000 7f4437fff700 1 -- 172.21.15.88:0/2531164054 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f44440095f0 con 0x7f44500f5410 2024-02-19T22:14:06.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:06.507+0000 7f4455ea6700 1 -- 172.21.15.88:0/2531164054 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4450045ad0 con 0x7f44500f5410 2024-02-19T22:14:06.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:06.507+0000 7f4437fff700 1 --2- 172.21.15.88:0/2531164054 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f443802ff10 0x7f44380323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:06.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:06.507+0000 7f4437fff700 1 -- 172.21.15.88:0/2531164054 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4444040e50 con 0x7f44500f5410 2024-02-19T22:14:06.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:06.507+0000 7f444effd700 1 --2- 172.21.15.88:0/2531164054 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f443802ff10 0x7f44380323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:06.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:06.507+0000 7f444effd700 1 --2- 172.21.15.88:0/2531164054 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f443802ff10 0x7f44380323d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f44400096a0 tx=0x7f4440006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:06.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:06.512+0000 7f4437fff700 1 -- 172.21.15.88:0/2531164054 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4444014420 con 0x7f44500f5410 2024-02-19T22:14:06.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:06.713+0000 7f4455ea6700 1 -- 172.21.15.88:0/2531164054 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f44500fafa0 con 0x7f44500f5410 2024-02-19T22:14:06.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:06.714+0000 7f4437fff700 1 -- 172.21.15.88:0/2531164054 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f444401d980 con 0x7f44500f5410 2024-02-19T22:14:06.716 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:14:06.716 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:14:06.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:06.716+0000 7f4455ea6700 1 -- 172.21.15.88:0/2531164054 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f443802ff10 msgr2=0x7f44380323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:06.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:06.716+0000 7f4455ea6700 1 --2- 172.21.15.88:0/2531164054 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f443802ff10 0x7f44380323d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f44400096a0 tx=0x7f4440006b40).stop 2024-02-19T22:14:06.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:06.716+0000 7f4455ea6700 1 -- 172.21.15.88:0/2531164054 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44500f5410 msgr2=0x7f44501089b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:06.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:06.716+0000 7f4455ea6700 1 --2- 172.21.15.88:0/2531164054 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44500f5410 0x7f44501089b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f4444005010 tx=0x7f444400b040).stop 2024-02-19T22:14:06.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:06.716+0000 7f4455ea6700 1 -- 172.21.15.88:0/2531164054 shutdown_connections 2024-02-19T22:14:06.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:06.716+0000 7f4455ea6700 1 --2- 172.21.15.88:0/2531164054 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f443802ff10 0x7f44380323d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:06.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:06.716+0000 7f4455ea6700 1 --2- 172.21.15.88:0/2531164054 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44500f5410 0x7f44501089b0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:06.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:06.716+0000 7f4455ea6700 1 -- 172.21.15.88:0/2531164054 >> 172.21.15.88:0/2531164054 conn(0x7f44500f0b70 msgr2=0x7f445005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:06.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:06.716+0000 7f4455ea6700 1 -- 172.21.15.88:0/2531164054 shutdown_connections 2024-02-19T22:14:06.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:06.716+0000 7f4455ea6700 1 -- 172.21.15.88:0/2531164054 wait complete. 2024-02-19T22:14:06.719 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:14:07.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:07 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:14:05.657080+0000 mgr.smithi052.ghqrtk (mgr.14162) 132 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:07.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:07 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:14:06.715690+0000 mon.smithi052 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.88:0/2531164054' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:08.340 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:08.341 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:09.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:09 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:14:07.657367+0000 mgr.smithi052.ghqrtk (mgr.14162) 133 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:10.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:10.201+0000 7fc1d76c0700 1 -- 172.21.15.88:0/1450849216 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc1d00f5410 msgr2=0x7fc1d00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:10.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:10.201+0000 7fc1d76c0700 1 --2- 172.21.15.88:0/1450849216 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc1d00f5410 0x7fc1d00f5830 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7fc1c0009fb0 tx=0x7fc1c0009b30).stop 2024-02-19T22:14:10.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:10.201+0000 7fc1d76c0700 1 -- 172.21.15.88:0/1450849216 shutdown_connections 2024-02-19T22:14:10.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:10.201+0000 7fc1d76c0700 1 --2- 172.21.15.88:0/1450849216 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc1d00f5410 0x7fc1d00f5830 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:10.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:10.201+0000 7fc1d76c0700 1 -- 172.21.15.88:0/1450849216 >> 172.21.15.88:0/1450849216 conn(0x7fc1d00f0b70 msgr2=0x7fc1d00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:10.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:10.201+0000 7fc1d76c0700 1 -- 172.21.15.88:0/1450849216 shutdown_connections 2024-02-19T22:14:10.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:10.201+0000 7fc1d76c0700 1 -- 172.21.15.88:0/1450849216 wait complete. 2024-02-19T22:14:10.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:10.201+0000 7fc1d76c0700 1 Processor -- start 2024-02-19T22:14:10.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:10.201+0000 7fc1d76c0700 1 -- start start 2024-02-19T22:14:10.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:10.202+0000 7fc1d76c0700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc1d00f5410 0x7fc1d01094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:10.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:10.202+0000 7fc1d76c0700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d0060240 con 0x7fc1d00f5410 2024-02-19T22:14:10.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:10.202+0000 7fc1d545c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc1d00f5410 0x7fc1d01094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:10.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:10.202+0000 7fc1d545c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc1d00f5410 0x7fc1d01094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:38938/0 (socket says 172.21.15.88:38938) 2024-02-19T22:14:10.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:10.202+0000 7fc1d545c700 1 -- 172.21.15.88:0/3294378965 learned_addr learned my addr 172.21.15.88:0/3294378965 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:14:10.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:10.202+0000 7fc1d545c700 1 -- 172.21.15.88:0/3294378965 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c0005040 con 0x7fc1d00f5410 2024-02-19T22:14:10.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:10.202+0000 7fc1d545c700 1 --2- 172.21.15.88:0/3294378965 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc1d00f5410 0x7fc1d01094e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fc1c0005010 tx=0x7fc1c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:10.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:10.202+0000 7fc1c5ffb700 1 -- 172.21.15.88:0/3294378965 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1c0009320 con 0x7fc1d00f5410 2024-02-19T22:14:10.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:10.202+0000 7fc1d76c0700 1 -- 172.21.15.88:0/3294378965 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1d0060440 con 0x7fc1d00f5410 2024-02-19T22:14:10.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:10.202+0000 7fc1c5ffb700 1 -- 172.21.15.88:0/3294378965 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc1c0013380 con 0x7fc1d00f5410 2024-02-19T22:14:10.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:10.202+0000 7fc1c5ffb700 1 -- 172.21.15.88:0/3294378965 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1c0014e50 con 0x7fc1d00f5410 2024-02-19T22:14:10.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:10.203+0000 7fc1d76c0700 1 -- 172.21.15.88:0/3294378965 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1d0109b90 con 0x7fc1d00f5410 2024-02-19T22:14:10.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:10.203+0000 7fc1c5ffb700 1 -- 172.21.15.88:0/3294378965 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc1c00095f0 con 0x7fc1d00f5410 2024-02-19T22:14:10.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:10.203+0000 7fc1d76c0700 1 -- 172.21.15.88:0/3294378965 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1d0045ad0 con 0x7fc1d00f5410 2024-02-19T22:14:10.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:10.204+0000 7fc1c5ffb700 1 --2- 172.21.15.88:0/3294378965 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fc1bc02ff10 0x7fc1bc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:10.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:10.204+0000 7fc1c5ffb700 1 -- 172.21.15.88:0/3294378965 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc1c0040f80 con 0x7fc1d00f5410 2024-02-19T22:14:10.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:10.204+0000 7fc1d4c5b700 1 --2- 172.21.15.88:0/3294378965 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fc1bc02ff10 0x7fc1bc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:10.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:10.204+0000 7fc1d4c5b700 1 --2- 172.21.15.88:0/3294378965 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fc1bc02ff10 0x7fc1bc0323d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fc1cc0096a0 tx=0x7fc1cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:10.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:10.208+0000 7fc1c5ffb700 1 -- 172.21.15.88:0/3294378965 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc1c0014420 con 0x7fc1d00f5410 2024-02-19T22:14:10.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:10.413+0000 7fc1d76c0700 1 -- 172.21.15.88:0/3294378965 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc1d010a3a0 con 0x7fc1d00f5410 2024-02-19T22:14:10.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:10.414+0000 7fc1c5ffb700 1 -- 172.21.15.88:0/3294378965 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc1c0010230 con 0x7fc1d00f5410 2024-02-19T22:14:10.416 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:14:10.416 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:14:10.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:10.416+0000 7fc1d76c0700 1 -- 172.21.15.88:0/3294378965 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fc1bc02ff10 msgr2=0x7fc1bc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:10.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:10.416+0000 7fc1d76c0700 1 --2- 172.21.15.88:0/3294378965 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fc1bc02ff10 0x7fc1bc0323d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fc1cc0096a0 tx=0x7fc1cc006b40).stop 2024-02-19T22:14:10.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:10.416+0000 7fc1d76c0700 1 -- 172.21.15.88:0/3294378965 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc1d00f5410 msgr2=0x7fc1d01094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:10.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:10.416+0000 7fc1d76c0700 1 --2- 172.21.15.88:0/3294378965 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc1d00f5410 0x7fc1d01094e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fc1c0005010 tx=0x7fc1c000b040).stop 2024-02-19T22:14:10.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:10.416+0000 7fc1d76c0700 1 -- 172.21.15.88:0/3294378965 shutdown_connections 2024-02-19T22:14:10.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:10.416+0000 7fc1d76c0700 1 --2- 172.21.15.88:0/3294378965 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fc1bc02ff10 0x7fc1bc0323d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:10.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:10.416+0000 7fc1d76c0700 1 --2- 172.21.15.88:0/3294378965 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc1d00f5410 0x7fc1d01094e0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:10.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:10.416+0000 7fc1d76c0700 1 -- 172.21.15.88:0/3294378965 >> 172.21.15.88:0/3294378965 conn(0x7fc1d00f0b70 msgr2=0x7fc1d00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:10.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:10.416+0000 7fc1d76c0700 1 -- 172.21.15.88:0/3294378965 shutdown_connections 2024-02-19T22:14:10.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:10.416+0000 7fc1d76c0700 1 -- 172.21.15.88:0/3294378965 wait complete. 2024-02-19T22:14:10.419 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:14:11.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:11 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:14:09.657628+0000 mgr.smithi052.ghqrtk (mgr.14162) 134 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:11.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:11 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:14:10.415492+0000 mon.smithi052 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.88:0/3294378965' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:12.085 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:12.085 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:13.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:13 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:14:11.657963+0000 mgr.smithi052.ghqrtk (mgr.14162) 135 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:13.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:13.926+0000 7fd463fff700 1 -- 172.21.15.88:0/1849843237 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd4640f78e0 msgr2=0x7fd4640f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:13.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:13.926+0000 7fd463fff700 1 --2- 172.21.15.88:0/1849843237 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd4640f78e0 0x7fd4640f7d00 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7fd44c009fb0 tx=0x7fd44c009b30).stop 2024-02-19T22:14:13.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:13.926+0000 7fd463fff700 1 -- 172.21.15.88:0/1849843237 shutdown_connections 2024-02-19T22:14:13.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:13.926+0000 7fd463fff700 1 --2- 172.21.15.88:0/1849843237 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd4640f78e0 0x7fd4640f7d00 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:13.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:13.926+0000 7fd463fff700 1 -- 172.21.15.88:0/1849843237 >> 172.21.15.88:0/1849843237 conn(0x7fd4640f0120 msgr2=0x7fd4640f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:13.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:13.926+0000 7fd463fff700 1 -- 172.21.15.88:0/1849843237 shutdown_connections 2024-02-19T22:14:13.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:13.926+0000 7fd463fff700 1 -- 172.21.15.88:0/1849843237 wait complete. 2024-02-19T22:14:13.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:13.927+0000 7fd463fff700 1 Processor -- start 2024-02-19T22:14:13.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:13.927+0000 7fd463fff700 1 -- start start 2024-02-19T22:14:13.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:13.927+0000 7fd463fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd4640f78e0 0x7fd4641065e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:13.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:13.927+0000 7fd463fff700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd464106ad0 con 0x7fd4640f78e0 2024-02-19T22:14:13.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:13.927+0000 7fd462ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd4640f78e0 0x7fd4641065e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:13.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:13.927+0000 7fd462ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd4640f78e0 0x7fd4641065e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:49378/0 (socket says 172.21.15.88:49378) 2024-02-19T22:14:13.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:13.927+0000 7fd462ffd700 1 -- 172.21.15.88:0/1797169461 learned_addr learned my addr 172.21.15.88:0/1797169461 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:14:13.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:13.928+0000 7fd462ffd700 1 -- 172.21.15.88:0/1797169461 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd44c005040 con 0x7fd4640f78e0 2024-02-19T22:14:13.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:13.928+0000 7fd462ffd700 1 --2- 172.21.15.88:0/1797169461 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd4640f78e0 0x7fd4641065e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fd44c0089f0 tx=0x7fd44c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:13.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:13.928+0000 7fd45b7fe700 1 -- 172.21.15.88:0/1797169461 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd44c009320 con 0x7fd4640f78e0 2024-02-19T22:14:13.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:13.928+0000 7fd463fff700 1 -- 172.21.15.88:0/1797169461 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd464106cd0 con 0x7fd4640f78e0 2024-02-19T22:14:13.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:13.928+0000 7fd45b7fe700 1 -- 172.21.15.88:0/1797169461 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd44c013380 con 0x7fd4640f78e0 2024-02-19T22:14:13.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:13.928+0000 7fd45b7fe700 1 -- 172.21.15.88:0/1797169461 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd44c014e20 con 0x7fd4640f78e0 2024-02-19T22:14:13.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:13.928+0000 7fd463fff700 1 -- 172.21.15.88:0/1797169461 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd464107160 con 0x7fd4640f78e0 2024-02-19T22:14:13.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:13.929+0000 7fd463fff700 1 -- 172.21.15.88:0/1797169461 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd464045ad0 con 0x7fd4640f78e0 2024-02-19T22:14:13.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:13.929+0000 7fd45b7fe700 1 -- 172.21.15.88:0/1797169461 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd44c0095f0 con 0x7fd4640f78e0 2024-02-19T22:14:13.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:13.929+0000 7fd45b7fe700 1 --2- 172.21.15.88:0/1797169461 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd45002ff10 0x7fd4500323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:13.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:13.929+0000 7fd45b7fe700 1 -- 172.21.15.88:0/1797169461 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd44c040ec0 con 0x7fd4640f78e0 2024-02-19T22:14:13.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:13.929+0000 7fd4627fc700 1 --2- 172.21.15.88:0/1797169461 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd45002ff10 0x7fd4500323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:13.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:13.929+0000 7fd4627fc700 1 --2- 172.21.15.88:0/1797169461 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd45002ff10 0x7fd4500323d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fd4540096a0 tx=0x7fd454006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:13.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:13.934+0000 7fd45b7fe700 1 -- 172.21.15.88:0/1797169461 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd44c014420 con 0x7fd4640f78e0 2024-02-19T22:14:14.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:14.137+0000 7fd463fff700 1 -- 172.21.15.88:0/1797169461 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd464108010 con 0x7fd4640f78e0 2024-02-19T22:14:14.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:14.138+0000 7fd45b7fe700 1 -- 172.21.15.88:0/1797169461 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd44c010230 con 0x7fd4640f78e0 2024-02-19T22:14:14.141 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:14:14.141 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:14:14.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:14.139+0000 7fd463fff700 1 -- 172.21.15.88:0/1797169461 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd45002ff10 msgr2=0x7fd4500323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:14.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:14.139+0000 7fd463fff700 1 --2- 172.21.15.88:0/1797169461 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd45002ff10 0x7fd4500323d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fd4540096a0 tx=0x7fd454006b40).stop 2024-02-19T22:14:14.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:14.139+0000 7fd463fff700 1 -- 172.21.15.88:0/1797169461 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd4640f78e0 msgr2=0x7fd4641065e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:14.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:14.139+0000 7fd463fff700 1 --2- 172.21.15.88:0/1797169461 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd4640f78e0 0x7fd4641065e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fd44c0089f0 tx=0x7fd44c00b040).stop 2024-02-19T22:14:14.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:14.139+0000 7fd463fff700 1 -- 172.21.15.88:0/1797169461 shutdown_connections 2024-02-19T22:14:14.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:14.139+0000 7fd463fff700 1 --2- 172.21.15.88:0/1797169461 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd45002ff10 0x7fd4500323d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:14.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:14.139+0000 7fd463fff700 1 --2- 172.21.15.88:0/1797169461 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd4640f78e0 0x7fd4641065e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:14.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:14.139+0000 7fd463fff700 1 -- 172.21.15.88:0/1797169461 >> 172.21.15.88:0/1797169461 conn(0x7fd4640f0120 msgr2=0x7fd4640f0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:14.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:14.140+0000 7fd463fff700 1 -- 172.21.15.88:0/1797169461 shutdown_connections 2024-02-19T22:14:14.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:14.140+0000 7fd463fff700 1 -- 172.21.15.88:0/1797169461 wait complete. 2024-02-19T22:14:14.144 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:14:15.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:15 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:14:13.658193+0000 mgr.smithi052.ghqrtk (mgr.14162) 136 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:15.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:15 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:14:14.139393+0000 mon.smithi052 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.88:0/1797169461' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:15.657 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:15.657 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:17.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:17.428+0000 7f2f16cb8700 1 -- 172.21.15.88:0/1310214288 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2f10060220 msgr2=0x7f2f100f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:17.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:17.428+0000 7f2f16cb8700 1 --2- 172.21.15.88:0/1310214288 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2f10060220 0x7f2f100f9150 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f2f04009fb0 tx=0x7f2f04009b30).stop 2024-02-19T22:14:17.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:17.428+0000 7f2f16cb8700 1 -- 172.21.15.88:0/1310214288 shutdown_connections 2024-02-19T22:14:17.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:17.428+0000 7f2f16cb8700 1 --2- 172.21.15.88:0/1310214288 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2f10060220 0x7f2f100f9150 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:17.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:17.428+0000 7f2f16cb8700 1 -- 172.21.15.88:0/1310214288 >> 172.21.15.88:0/1310214288 conn(0x7f2f100f0b70 msgr2=0x7f2f100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:17.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:17.428+0000 7f2f16cb8700 1 -- 172.21.15.88:0/1310214288 shutdown_connections 2024-02-19T22:14:17.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:17.428+0000 7f2f16cb8700 1 -- 172.21.15.88:0/1310214288 wait complete. 2024-02-19T22:14:17.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:17.429+0000 7f2f16cb8700 1 Processor -- start 2024-02-19T22:14:17.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:17.429+0000 7f2f16cb8700 1 -- start start 2024-02-19T22:14:17.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:17.429+0000 7f2f16cb8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2f10060220 0x7f2f100f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:17.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:17.429+0000 7f2f16cb8700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f100f5050 con 0x7f2f10060220 2024-02-19T22:14:17.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:17.429+0000 7f2f14a54700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2f10060220 0x7f2f100f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:17.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:17.429+0000 7f2f14a54700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2f10060220 0x7f2f100f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:49398/0 (socket says 172.21.15.88:49398) 2024-02-19T22:14:17.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:17.429+0000 7f2f14a54700 1 -- 172.21.15.88:0/1383416918 learned_addr learned my addr 172.21.15.88:0/1383416918 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:14:17.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:17.429+0000 7f2f14a54700 1 -- 172.21.15.88:0/1383416918 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f04005040 con 0x7f2f10060220 2024-02-19T22:14:17.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:17.429+0000 7f2f14a54700 1 --2- 172.21.15.88:0/1383416918 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2f10060220 0x7f2f100f6bd0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f2f040089f0 tx=0x7f2f0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:17.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:17.430+0000 7f2f0d7fa700 1 -- 172.21.15.88:0/1383416918 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f04009320 con 0x7f2f10060220 2024-02-19T22:14:17.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:17.430+0000 7f2f16cb8700 1 -- 172.21.15.88:0/1383416918 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f100f51f0 con 0x7f2f10060220 2024-02-19T22:14:17.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:17.430+0000 7f2f0d7fa700 1 -- 172.21.15.88:0/1383416918 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2f04013380 con 0x7f2f10060220 2024-02-19T22:14:17.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:17.430+0000 7f2f0d7fa700 1 -- 172.21.15.88:0/1383416918 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f04014e20 con 0x7f2f10060220 2024-02-19T22:14:17.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:17.430+0000 7f2f16cb8700 1 -- 172.21.15.88:0/1383416918 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f100f5680 con 0x7f2f10060220 2024-02-19T22:14:17.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:17.430+0000 7f2f0d7fa700 1 -- 172.21.15.88:0/1383416918 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2f040263a0 con 0x7f2f10060220 2024-02-19T22:14:17.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:17.431+0000 7f2f0d7fa700 1 --2- 172.21.15.88:0/1383416918 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f2f0002ff10 0x7f2f000323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:17.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:17.431+0000 7f2f0d7fa700 1 -- 172.21.15.88:0/1383416918 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2f04040f60 con 0x7f2f10060220 2024-02-19T22:14:17.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:17.431+0000 7f2f16cb8700 1 -- 172.21.15.88:0/1383416918 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f10059100 con 0x7f2f10060220 2024-02-19T22:14:17.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:17.431+0000 7f2f0ffff700 1 --2- 172.21.15.88:0/1383416918 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f2f0002ff10 0x7f2f000323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:17.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:17.431+0000 7f2f0ffff700 1 --2- 172.21.15.88:0/1383416918 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f2f0002ff10 0x7f2f000323d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f2efc0096a0 tx=0x7f2efc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:17.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:17.436+0000 7f2f0d7fa700 1 -- 172.21.15.88:0/1383416918 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2f04012920 con 0x7f2f10060220 2024-02-19T22:14:17.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:17 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:14:15.658541+0000 mgr.smithi052.ghqrtk (mgr.14162) 137 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:17.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:17.640+0000 7f2f16cb8700 1 -- 172.21.15.88:0/1383416918 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2f10045ad0 con 0x7f2f10060220 2024-02-19T22:14:17.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:17.640+0000 7f2f0d7fa700 1 -- 172.21.15.88:0/1383416918 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2f10045ad0 con 0x7f2f10060220 2024-02-19T22:14:17.643 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:14:17.643 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:14:17.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:17.642+0000 7f2f16cb8700 1 -- 172.21.15.88:0/1383416918 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f2f0002ff10 msgr2=0x7f2f000323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:17.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:17.642+0000 7f2f16cb8700 1 --2- 172.21.15.88:0/1383416918 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f2f0002ff10 0x7f2f000323d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f2efc0096a0 tx=0x7f2efc006b40).stop 2024-02-19T22:14:17.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:17.642+0000 7f2f16cb8700 1 -- 172.21.15.88:0/1383416918 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2f10060220 msgr2=0x7f2f100f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:17.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:17.642+0000 7f2f16cb8700 1 --2- 172.21.15.88:0/1383416918 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2f10060220 0x7f2f100f6bd0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f2f040089f0 tx=0x7f2f0400b040).stop 2024-02-19T22:14:17.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:17.642+0000 7f2f16cb8700 1 -- 172.21.15.88:0/1383416918 shutdown_connections 2024-02-19T22:14:17.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:17.642+0000 7f2f16cb8700 1 --2- 172.21.15.88:0/1383416918 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f2f0002ff10 0x7f2f000323d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:17.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:17.642+0000 7f2f16cb8700 1 --2- 172.21.15.88:0/1383416918 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2f10060220 0x7f2f100f6bd0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:17.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:17.642+0000 7f2f16cb8700 1 -- 172.21.15.88:0/1383416918 >> 172.21.15.88:0/1383416918 conn(0x7f2f100f0b70 msgr2=0x7f2f100f1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:17.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:17.642+0000 7f2f16cb8700 1 -- 172.21.15.88:0/1383416918 shutdown_connections 2024-02-19T22:14:17.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:17.642+0000 7f2f16cb8700 1 -- 172.21.15.88:0/1383416918 wait complete. 2024-02-19T22:14:17.645 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:14:18.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:14:17.641852+0000 mon.smithi052 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.88:0/1383416918' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:19.304 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:19.304 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:19.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:19 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:14:17.658932+0000 mgr.smithi052.ghqrtk (mgr.14162) 138 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:21.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:21.127+0000 7f32121ab700 1 -- 172.21.15.88:0/2876211355 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f320c0f6510 msgr2=0x7f320c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:21.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:21.127+0000 7f32121ab700 1 --2- 172.21.15.88:0/2876211355 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f320c0f6510 0x7f320c0f6930 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f31f4009fb0 tx=0x7f31f4009b30).stop 2024-02-19T22:14:21.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:21.127+0000 7f32121ab700 1 -- 172.21.15.88:0/2876211355 shutdown_connections 2024-02-19T22:14:21.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:21.127+0000 7f32121ab700 1 --2- 172.21.15.88:0/2876211355 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f320c0f6510 0x7f320c0f6930 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:21.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:21.127+0000 7f32121ab700 1 -- 172.21.15.88:0/2876211355 >> 172.21.15.88:0/2876211355 conn(0x7f320c0f1c70 msgr2=0x7f320c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:21.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:21.127+0000 7f32121ab700 1 -- 172.21.15.88:0/2876211355 shutdown_connections 2024-02-19T22:14:21.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:21.127+0000 7f32121ab700 1 -- 172.21.15.88:0/2876211355 wait complete. 2024-02-19T22:14:21.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:21.128+0000 7f32121ab700 1 Processor -- start 2024-02-19T22:14:21.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:21.128+0000 7f32121ab700 1 -- start start 2024-02-19T22:14:21.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:21.128+0000 7f32121ab700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f320c0f6510 0x7f320c108a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:21.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:21.128+0000 7f32121ab700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f320c108ef0 con 0x7f320c0f6510 2024-02-19T22:14:21.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:21.128+0000 7f320b7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f320c0f6510 0x7f320c108a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:21.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:21.128+0000 7f320b7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f320c0f6510 0x7f320c108a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:49424/0 (socket says 172.21.15.88:49424) 2024-02-19T22:14:21.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:21.128+0000 7f320b7fe700 1 -- 172.21.15.88:0/3378779493 learned_addr learned my addr 172.21.15.88:0/3378779493 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:14:21.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:21.129+0000 7f320b7fe700 1 -- 172.21.15.88:0/3378779493 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f4005040 con 0x7f320c0f6510 2024-02-19T22:14:21.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:21.129+0000 7f320b7fe700 1 --2- 172.21.15.88:0/3378779493 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f320c0f6510 0x7f320c108a00 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f31f4005010 tx=0x7f31f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:21.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:21.129+0000 7f3203fff700 1 -- 172.21.15.88:0/3378779493 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f31f4009320 con 0x7f320c0f6510 2024-02-19T22:14:21.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:21.129+0000 7f32121ab700 1 -- 172.21.15.88:0/3378779493 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f320c1090f0 con 0x7f320c0f6510 2024-02-19T22:14:21.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:21.129+0000 7f3203fff700 1 -- 172.21.15.88:0/3378779493 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f31f4013380 con 0x7f320c0f6510 2024-02-19T22:14:21.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:21.129+0000 7f3203fff700 1 -- 172.21.15.88:0/3378779493 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f31f4014e50 con 0x7f320c0f6510 2024-02-19T22:14:21.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:21.129+0000 7f32121ab700 1 -- 172.21.15.88:0/3378779493 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f320c109580 con 0x7f320c0f6510 2024-02-19T22:14:21.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:21.130+0000 7f3203fff700 1 -- 172.21.15.88:0/3378779493 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f31f40263a0 con 0x7f320c0f6510 2024-02-19T22:14:21.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:21.130+0000 7f32121ab700 1 -- 172.21.15.88:0/3378779493 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f320c045ad0 con 0x7f320c0f6510 2024-02-19T22:14:21.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:21.130+0000 7f3203fff700 1 --2- 172.21.15.88:0/3378779493 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f31f802ff10 0x7f31f80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:21.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:21.130+0000 7f3203fff700 1 -- 172.21.15.88:0/3378779493 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f31f4040fd0 con 0x7f320c0f6510 2024-02-19T22:14:21.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:21.130+0000 7f320affd700 1 --2- 172.21.15.88:0/3378779493 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f31f802ff10 0x7f31f80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:21.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:21.130+0000 7f320affd700 1 --2- 172.21.15.88:0/3378779493 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f31f802ff10 0x7f31f80323d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f31fc0096a0 tx=0x7f31fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:21.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:21.135+0000 7f3203fff700 1 -- 172.21.15.88:0/3378779493 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f31f40126f0 con 0x7f320c0f6510 2024-02-19T22:14:21.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:21.337+0000 7f32121ab700 1 -- 172.21.15.88:0/3378779493 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f320c10a430 con 0x7f320c0f6510 2024-02-19T22:14:21.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:21.338+0000 7f3203fff700 1 -- 172.21.15.88:0/3378779493 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f31f40126f0 con 0x7f320c0f6510 2024-02-19T22:14:21.340 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:14:21.340 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:14:21.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:21.339+0000 7f32121ab700 1 -- 172.21.15.88:0/3378779493 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f31f802ff10 msgr2=0x7f31f80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:21.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:21.339+0000 7f32121ab700 1 --2- 172.21.15.88:0/3378779493 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f31f802ff10 0x7f31f80323d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f31fc0096a0 tx=0x7f31fc006b40).stop 2024-02-19T22:14:21.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:21.339+0000 7f32121ab700 1 -- 172.21.15.88:0/3378779493 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f320c0f6510 msgr2=0x7f320c108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:21.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:21.339+0000 7f32121ab700 1 --2- 172.21.15.88:0/3378779493 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f320c0f6510 0x7f320c108a00 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f31f4005010 tx=0x7f31f400b040).stop 2024-02-19T22:14:21.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:21.339+0000 7f32121ab700 1 -- 172.21.15.88:0/3378779493 shutdown_connections 2024-02-19T22:14:21.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:21.339+0000 7f32121ab700 1 --2- 172.21.15.88:0/3378779493 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f31f802ff10 0x7f31f80323d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:21.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:21.339+0000 7f32121ab700 1 --2- 172.21.15.88:0/3378779493 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f320c0f6510 0x7f320c108a00 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:21.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:21.339+0000 7f32121ab700 1 -- 172.21.15.88:0/3378779493 >> 172.21.15.88:0/3378779493 conn(0x7f320c0f1c70 msgr2=0x7f320c0f26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:21.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:21.339+0000 7f32121ab700 1 -- 172.21.15.88:0/3378779493 shutdown_connections 2024-02-19T22:14:21.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:21.339+0000 7f32121ab700 1 -- 172.21.15.88:0/3378779493 wait complete. 2024-02-19T22:14:21.342 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:14:21.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:21 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:14:19.659254+0000 mgr.smithi052.ghqrtk (mgr.14162) 139 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:22.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:22 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:14:21.339099+0000 mon.smithi052 (mon.0) 190 : audit [DBG] from='client.? 172.21.15.88:0/3378779493' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:22.980 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:22.980 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:23.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:23 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:14:21.659590+0000 mgr.smithi052.ghqrtk (mgr.14162) 140 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:24.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:24.827+0000 7f9b3693a700 1 -- 172.21.15.88:0/153245396 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b300f5410 msgr2=0x7f9b300f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:24.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:24.827+0000 7f9b3693a700 1 --2- 172.21.15.88:0/153245396 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b300f5410 0x7f9b300f5830 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f9b20009fb0 tx=0x7f9b20009b30).stop 2024-02-19T22:14:24.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:24.827+0000 7f9b3693a700 1 -- 172.21.15.88:0/153245396 shutdown_connections 2024-02-19T22:14:24.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:24.827+0000 7f9b3693a700 1 --2- 172.21.15.88:0/153245396 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b300f5410 0x7f9b300f5830 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:24.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:24.827+0000 7f9b3693a700 1 -- 172.21.15.88:0/153245396 >> 172.21.15.88:0/153245396 conn(0x7f9b300f0b70 msgr2=0x7f9b300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:24.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:24.827+0000 7f9b3693a700 1 -- 172.21.15.88:0/153245396 shutdown_connections 2024-02-19T22:14:24.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:24.827+0000 7f9b3693a700 1 -- 172.21.15.88:0/153245396 wait complete. 2024-02-19T22:14:24.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:24.828+0000 7f9b3693a700 1 Processor -- start 2024-02-19T22:14:24.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:24.828+0000 7f9b3693a700 1 -- start start 2024-02-19T22:14:24.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:24.828+0000 7f9b3693a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b300f5410 0x7f9b301094a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:24.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:24.828+0000 7f9b3693a700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b30060280 con 0x7f9b300f5410 2024-02-19T22:14:24.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:24.829+0000 7f9b2ffff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b300f5410 0x7f9b301094a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:24.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:24.829+0000 7f9b2ffff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b300f5410 0x7f9b301094a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:48258/0 (socket says 172.21.15.88:48258) 2024-02-19T22:14:24.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:24.829+0000 7f9b2ffff700 1 -- 172.21.15.88:0/2828936733 learned_addr learned my addr 172.21.15.88:0/2828936733 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:14:24.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:24.829+0000 7f9b2ffff700 1 -- 172.21.15.88:0/2828936733 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b20005040 con 0x7f9b300f5410 2024-02-19T22:14:24.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:24.829+0000 7f9b2ffff700 1 --2- 172.21.15.88:0/2828936733 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b300f5410 0x7f9b301094a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f9b200089f0 tx=0x7f9b2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:24.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:24.829+0000 7f9b2cff9700 1 -- 172.21.15.88:0/2828936733 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b20009320 con 0x7f9b300f5410 2024-02-19T22:14:24.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:24.829+0000 7f9b2cff9700 1 -- 172.21.15.88:0/2828936733 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9b20013380 con 0x7f9b300f5410 2024-02-19T22:14:24.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:24.829+0000 7f9b3693a700 1 -- 172.21.15.88:0/2828936733 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b30060480 con 0x7f9b300f5410 2024-02-19T22:14:24.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:24.829+0000 7f9b2cff9700 1 -- 172.21.15.88:0/2828936733 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b2001ed80 con 0x7f9b300f5410 2024-02-19T22:14:24.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:24.829+0000 7f9b3693a700 1 -- 172.21.15.88:0/2828936733 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b30109bd0 con 0x7f9b300f5410 2024-02-19T22:14:24.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:24.830+0000 7f9b2cff9700 1 -- 172.21.15.88:0/2828936733 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9b2001e420 con 0x7f9b300f5410 2024-02-19T22:14:24.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:24.830+0000 7f9b3693a700 1 -- 172.21.15.88:0/2828936733 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b30045ad0 con 0x7f9b300f5410 2024-02-19T22:14:24.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:24.830+0000 7f9b2cff9700 1 --2- 172.21.15.88:0/2828936733 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f9b1802ff60 0x7f9b18032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:24.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:24.830+0000 7f9b2cff9700 1 -- 172.21.15.88:0/2828936733 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9b20040ed0 con 0x7f9b300f5410 2024-02-19T22:14:24.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:24.830+0000 7f9b2f7fe700 1 --2- 172.21.15.88:0/2828936733 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f9b1802ff60 0x7f9b18032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:24.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:24.831+0000 7f9b2f7fe700 1 --2- 172.21.15.88:0/2828936733 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f9b1802ff60 0x7f9b18032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f9b240096a0 tx=0x7f9b24006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:24.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:24.835+0000 7f9b2cff9700 1 -- 172.21.15.88:0/2828936733 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9b2001cb40 con 0x7f9b300f5410 2024-02-19T22:14:25.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:25.042+0000 7f9b3693a700 1 -- 172.21.15.88:0/2828936733 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9b300fafa0 con 0x7f9b300f5410 2024-02-19T22:14:25.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:25.043+0000 7f9b2cff9700 1 -- 172.21.15.88:0/2828936733 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9b2001cb40 con 0x7f9b300f5410 2024-02-19T22:14:25.045 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:14:25.045 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:14:25.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:25.045+0000 7f9b3693a700 1 -- 172.21.15.88:0/2828936733 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f9b1802ff60 msgr2=0x7f9b18032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:25.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:25.045+0000 7f9b3693a700 1 --2- 172.21.15.88:0/2828936733 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f9b1802ff60 0x7f9b18032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f9b240096a0 tx=0x7f9b24006b40).stop 2024-02-19T22:14:25.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:25.045+0000 7f9b3693a700 1 -- 172.21.15.88:0/2828936733 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b300f5410 msgr2=0x7f9b301094a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:25.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:25.045+0000 7f9b3693a700 1 --2- 172.21.15.88:0/2828936733 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b300f5410 0x7f9b301094a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f9b200089f0 tx=0x7f9b2000b040).stop 2024-02-19T22:14:25.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:25.045+0000 7f9b3693a700 1 -- 172.21.15.88:0/2828936733 shutdown_connections 2024-02-19T22:14:25.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:25.045+0000 7f9b3693a700 1 --2- 172.21.15.88:0/2828936733 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f9b1802ff60 0x7f9b18032420 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:25.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:25.045+0000 7f9b3693a700 1 --2- 172.21.15.88:0/2828936733 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b300f5410 0x7f9b301094a0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:25.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:25.045+0000 7f9b3693a700 1 -- 172.21.15.88:0/2828936733 >> 172.21.15.88:0/2828936733 conn(0x7f9b300f0b70 msgr2=0x7f9b300f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:25.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:25.045+0000 7f9b3693a700 1 -- 172.21.15.88:0/2828936733 shutdown_connections 2024-02-19T22:14:25.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:25.045+0000 7f9b3693a700 1 -- 172.21.15.88:0/2828936733 wait complete. 2024-02-19T22:14:25.048 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:14:25.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:25 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:14:23.659908+0000 mgr.smithi052.ghqrtk (mgr.14162) 141 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:25.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:25 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:14:25.044302+0000 mon.smithi052 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.88:0/2828936733' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:26.689 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:26.689 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:27.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:27 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:14:25.660223+0000 mgr.smithi052.ghqrtk (mgr.14162) 142 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:28.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:28.477+0000 7f5783667700 1 -- 172.21.15.88:0/3399058225 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f577c0f5410 msgr2=0x7f577c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:28.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:28.477+0000 7f5783667700 1 --2- 172.21.15.88:0/3399058225 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f577c0f5410 0x7f577c0f5830 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f576c009fb0 tx=0x7f576c009b30).stop 2024-02-19T22:14:28.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:28.477+0000 7f5783667700 1 -- 172.21.15.88:0/3399058225 shutdown_connections 2024-02-19T22:14:28.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:28.477+0000 7f5783667700 1 --2- 172.21.15.88:0/3399058225 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f577c0f5410 0x7f577c0f5830 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:28.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:28.477+0000 7f5783667700 1 -- 172.21.15.88:0/3399058225 >> 172.21.15.88:0/3399058225 conn(0x7f577c0f0b70 msgr2=0x7f577c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:28.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:28.478+0000 7f5783667700 1 -- 172.21.15.88:0/3399058225 shutdown_connections 2024-02-19T22:14:28.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:28.478+0000 7f5783667700 1 -- 172.21.15.88:0/3399058225 wait complete. 2024-02-19T22:14:28.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:28.478+0000 7f5783667700 1 Processor -- start 2024-02-19T22:14:28.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:28.478+0000 7f5783667700 1 -- start start 2024-02-19T22:14:28.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:28.478+0000 7f5783667700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f577c0f5410 0x7f577c109540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:28.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:28.478+0000 7f5783667700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f577c060220 con 0x7f577c0f5410 2024-02-19T22:14:28.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:28.478+0000 7f5781403700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f577c0f5410 0x7f577c109540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:28.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:28.478+0000 7f5781403700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f577c0f5410 0x7f577c109540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:48270/0 (socket says 172.21.15.88:48270) 2024-02-19T22:14:28.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:28.478+0000 7f5781403700 1 -- 172.21.15.88:0/4137605950 learned_addr learned my addr 172.21.15.88:0/4137605950 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:14:28.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:28.479+0000 7f5781403700 1 -- 172.21.15.88:0/4137605950 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f576c005040 con 0x7f577c0f5410 2024-02-19T22:14:28.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:28.479+0000 7f5781403700 1 --2- 172.21.15.88:0/4137605950 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f577c0f5410 0x7f577c109540 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f576c005010 tx=0x7f576c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:28.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:28.479+0000 7f5771ffb700 1 -- 172.21.15.88:0/4137605950 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f576c009320 con 0x7f577c0f5410 2024-02-19T22:14:28.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:28.479+0000 7f5783667700 1 -- 172.21.15.88:0/4137605950 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f577c060420 con 0x7f577c0f5410 2024-02-19T22:14:28.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:28.479+0000 7f5771ffb700 1 -- 172.21.15.88:0/4137605950 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f576c013380 con 0x7f577c0f5410 2024-02-19T22:14:28.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:28.479+0000 7f5771ffb700 1 -- 172.21.15.88:0/4137605950 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f576c014e50 con 0x7f577c0f5410 2024-02-19T22:14:28.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:28.479+0000 7f5783667700 1 -- 172.21.15.88:0/4137605950 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f577c109bf0 con 0x7f577c0f5410 2024-02-19T22:14:28.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:28.480+0000 7f5771ffb700 1 -- 172.21.15.88:0/4137605950 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f576c0095f0 con 0x7f577c0f5410 2024-02-19T22:14:28.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:28.480+0000 7f5783667700 1 -- 172.21.15.88:0/4137605950 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f577c059100 con 0x7f577c0f5410 2024-02-19T22:14:28.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:28.480+0000 7f5771ffb700 1 --2- 172.21.15.88:0/4137605950 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f576802ff10 0x7f57680323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:28.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:28.480+0000 7f5771ffb700 1 -- 172.21.15.88:0/4137605950 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f576c040e90 con 0x7f577c0f5410 2024-02-19T22:14:28.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:28.480+0000 7f5780c02700 1 --2- 172.21.15.88:0/4137605950 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f576802ff10 0x7f57680323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:28.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:28.481+0000 7f5780c02700 1 --2- 172.21.15.88:0/4137605950 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f576802ff10 0x7f57680323d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f57780096a0 tx=0x7f5778006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:28.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:28.485+0000 7f5771ffb700 1 -- 172.21.15.88:0/4137605950 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f576c014420 con 0x7f577c0f5410 2024-02-19T22:14:28.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:28.690+0000 7f5783667700 1 -- 172.21.15.88:0/4137605950 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f577c045ad0 con 0x7f577c0f5410 2024-02-19T22:14:28.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:28.691+0000 7f5771ffb700 1 -- 172.21.15.88:0/4137605950 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f576c010230 con 0x7f577c0f5410 2024-02-19T22:14:28.693 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:14:28.693 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:14:28.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:28.692+0000 7f5783667700 1 -- 172.21.15.88:0/4137605950 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f576802ff10 msgr2=0x7f57680323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:28.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:28.692+0000 7f5783667700 1 --2- 172.21.15.88:0/4137605950 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f576802ff10 0x7f57680323d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f57780096a0 tx=0x7f5778006b40).stop 2024-02-19T22:14:28.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:28.692+0000 7f5783667700 1 -- 172.21.15.88:0/4137605950 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f577c0f5410 msgr2=0x7f577c109540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:28.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:28.692+0000 7f5783667700 1 --2- 172.21.15.88:0/4137605950 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f577c0f5410 0x7f577c109540 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f576c005010 tx=0x7f576c00b040).stop 2024-02-19T22:14:28.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:28.692+0000 7f5783667700 1 -- 172.21.15.88:0/4137605950 shutdown_connections 2024-02-19T22:14:28.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:28.692+0000 7f5783667700 1 --2- 172.21.15.88:0/4137605950 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f576802ff10 0x7f57680323d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:28.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:28.692+0000 7f5783667700 1 --2- 172.21.15.88:0/4137605950 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f577c0f5410 0x7f577c109540 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:28.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:28.692+0000 7f5783667700 1 -- 172.21.15.88:0/4137605950 >> 172.21.15.88:0/4137605950 conn(0x7f577c0f0b70 msgr2=0x7f577c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:28.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:28.692+0000 7f5783667700 1 -- 172.21.15.88:0/4137605950 shutdown_connections 2024-02-19T22:14:28.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:28.692+0000 7f5783667700 1 -- 172.21.15.88:0/4137605950 wait complete. 2024-02-19T22:14:28.695 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:14:29.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:29 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:14:27.660581+0000 mgr.smithi052.ghqrtk (mgr.14162) 143 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:29.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:29 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:14:28.692069+0000 mon.smithi052 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.88:0/4137605950' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:30.358 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:30.358 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:31.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:31 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:14:29.660943+0000 mgr.smithi052.ghqrtk (mgr.14162) 144 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:32.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:32.186+0000 7f594e5f7700 1 -- 172.21.15.88:0/2752931659 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5948060280 msgr2=0x7f59480f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:32.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:32.186+0000 7f594e5f7700 1 --2- 172.21.15.88:0/2752931659 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5948060280 0x7f59480f90f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f5930009fb0 tx=0x7f5930009b30).stop 2024-02-19T22:14:32.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:32.186+0000 7f594e5f7700 1 -- 172.21.15.88:0/2752931659 shutdown_connections 2024-02-19T22:14:32.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:32.186+0000 7f594e5f7700 1 --2- 172.21.15.88:0/2752931659 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5948060280 0x7f59480f90f0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:32.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:32.186+0000 7f594e5f7700 1 -- 172.21.15.88:0/2752931659 >> 172.21.15.88:0/2752931659 conn(0x7f59480f0b70 msgr2=0x7f59480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:32.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:32.186+0000 7f594e5f7700 1 -- 172.21.15.88:0/2752931659 shutdown_connections 2024-02-19T22:14:32.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:32.186+0000 7f594e5f7700 1 -- 172.21.15.88:0/2752931659 wait complete. 2024-02-19T22:14:32.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:32.187+0000 7f594e5f7700 1 Processor -- start 2024-02-19T22:14:32.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:32.187+0000 7f594e5f7700 1 -- start start 2024-02-19T22:14:32.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:32.187+0000 7f594e5f7700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5948060280 0x7f59480f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:32.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:32.187+0000 7f594e5f7700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59480f4ff0 con 0x7f5948060280 2024-02-19T22:14:32.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:32.187+0000 7f5947fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5948060280 0x7f59480f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:32.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:32.187+0000 7f5947fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5948060280 0x7f59480f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:48282/0 (socket says 172.21.15.88:48282) 2024-02-19T22:14:32.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:32.187+0000 7f5947fff700 1 -- 172.21.15.88:0/2340579354 learned_addr learned my addr 172.21.15.88:0/2340579354 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:14:32.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:32.187+0000 7f5947fff700 1 -- 172.21.15.88:0/2340579354 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5930005040 con 0x7f5948060280 2024-02-19T22:14:32.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:32.188+0000 7f5947fff700 1 --2- 172.21.15.88:0/2340579354 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5948060280 0x7f59480f6b70 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f5930009830 tx=0x7f593000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:32.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:32.188+0000 7f5944ff9700 1 -- 172.21.15.88:0/2340579354 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5930009320 con 0x7f5948060280 2024-02-19T22:14:32.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:32.188+0000 7f594e5f7700 1 -- 172.21.15.88:0/2340579354 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59480f5190 con 0x7f5948060280 2024-02-19T22:14:32.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:32.188+0000 7f5944ff9700 1 -- 172.21.15.88:0/2340579354 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5930013380 con 0x7f5948060280 2024-02-19T22:14:32.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:32.188+0000 7f5944ff9700 1 -- 172.21.15.88:0/2340579354 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5930014420 con 0x7f5948060280 2024-02-19T22:14:32.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:32.188+0000 7f594e5f7700 1 -- 172.21.15.88:0/2340579354 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59480f5620 con 0x7f5948060280 2024-02-19T22:14:32.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:32.189+0000 7f5944ff9700 1 -- 172.21.15.88:0/2340579354 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5930014580 con 0x7f5948060280 2024-02-19T22:14:32.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:32.189+0000 7f5944ff9700 1 --2- 172.21.15.88:0/2340579354 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f593402ff60 0x7f5934032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:32.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:32.189+0000 7f594e5f7700 1 -- 172.21.15.88:0/2340579354 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5948059100 con 0x7f5948060280 2024-02-19T22:14:32.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:32.189+0000 7f5944ff9700 1 -- 172.21.15.88:0/2340579354 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5930040e40 con 0x7f5948060280 2024-02-19T22:14:32.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:32.189+0000 7f59477fe700 1 --2- 172.21.15.88:0/2340579354 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f593402ff60 0x7f5934032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:32.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:32.189+0000 7f59477fe700 1 --2- 172.21.15.88:0/2340579354 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f593402ff60 0x7f5934032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f59380096a0 tx=0x7f5938006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:32.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:32.194+0000 7f5944ff9700 1 -- 172.21.15.88:0/2340579354 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5930014810 con 0x7f5948060280 2024-02-19T22:14:32.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:32.399+0000 7f594e5f7700 1 -- 172.21.15.88:0/2340579354 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5948045ad0 con 0x7f5948060280 2024-02-19T22:14:32.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:32.400+0000 7f5944ff9700 1 -- 172.21.15.88:0/2340579354 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5930004490 con 0x7f5948060280 2024-02-19T22:14:32.402 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:14:32.402 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:14:32.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:32.401+0000 7f594e5f7700 1 -- 172.21.15.88:0/2340579354 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f593402ff60 msgr2=0x7f5934032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:32.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:32.401+0000 7f594e5f7700 1 --2- 172.21.15.88:0/2340579354 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f593402ff60 0x7f5934032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f59380096a0 tx=0x7f5938006b40).stop 2024-02-19T22:14:32.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:32.401+0000 7f594e5f7700 1 -- 172.21.15.88:0/2340579354 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5948060280 msgr2=0x7f59480f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:32.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:32.401+0000 7f594e5f7700 1 --2- 172.21.15.88:0/2340579354 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5948060280 0x7f59480f6b70 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f5930009830 tx=0x7f593000b040).stop 2024-02-19T22:14:32.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:32.401+0000 7f594e5f7700 1 -- 172.21.15.88:0/2340579354 shutdown_connections 2024-02-19T22:14:32.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:32.401+0000 7f594e5f7700 1 --2- 172.21.15.88:0/2340579354 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f593402ff60 0x7f5934032420 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:32.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:32.401+0000 7f594e5f7700 1 --2- 172.21.15.88:0/2340579354 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5948060280 0x7f59480f6b70 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:32.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:32.401+0000 7f594e5f7700 1 -- 172.21.15.88:0/2340579354 >> 172.21.15.88:0/2340579354 conn(0x7f59480f0b70 msgr2=0x7f59480f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:32.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:32.402+0000 7f594e5f7700 1 -- 172.21.15.88:0/2340579354 shutdown_connections 2024-02-19T22:14:32.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:32.402+0000 7f594e5f7700 1 -- 172.21.15.88:0/2340579354 wait complete. 2024-02-19T22:14:32.405 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:14:33.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:33 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:14:31.661337+0000 mgr.smithi052.ghqrtk (mgr.14162) 145 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:33.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:33 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:14:32.401471+0000 mon.smithi052 (mon.0) 193 : audit [DBG] from='client.? 172.21.15.88:0/2340579354' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:34.072 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:34.073 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:35.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:35 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:14:33.661637+0000 mgr.smithi052.ghqrtk (mgr.14162) 146 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:35.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:35.798+0000 7f36d7023700 1 -- 172.21.15.88:0/193315936 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f36d0060200 msgr2=0x7f36d00fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:35.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:35.798+0000 7f36d7023700 1 --2- 172.21.15.88:0/193315936 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f36d0060200 0x7f36d00fb570 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f36c4009fb0 tx=0x7f36c4009b30).stop 2024-02-19T22:14:35.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:35.798+0000 7f36d7023700 1 -- 172.21.15.88:0/193315936 shutdown_connections 2024-02-19T22:14:35.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:35.798+0000 7f36d7023700 1 --2- 172.21.15.88:0/193315936 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f36d0060200 0x7f36d00fb570 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:35.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:35.798+0000 7f36d7023700 1 -- 172.21.15.88:0/193315936 >> 172.21.15.88:0/193315936 conn(0x7f36d00f0b70 msgr2=0x7f36d00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:35.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:35.798+0000 7f36d7023700 1 -- 172.21.15.88:0/193315936 shutdown_connections 2024-02-19T22:14:35.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:35.798+0000 7f36d7023700 1 -- 172.21.15.88:0/193315936 wait complete. 2024-02-19T22:14:35.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:35.799+0000 7f36d7023700 1 Processor -- start 2024-02-19T22:14:35.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:35.799+0000 7f36d7023700 1 -- start start 2024-02-19T22:14:35.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:35.799+0000 7f36d7023700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f36d0060200 0x7f36d01089a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:35.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:35.799+0000 7f36d7023700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d0108e90 con 0x7f36d0060200 2024-02-19T22:14:35.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:35.799+0000 7f36d4dbf700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f36d0060200 0x7f36d01089a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:35.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:35.799+0000 7f36d4dbf700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f36d0060200 0x7f36d01089a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:36990/0 (socket says 172.21.15.88:36990) 2024-02-19T22:14:35.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:35.799+0000 7f36d4dbf700 1 -- 172.21.15.88:0/3422793919 learned_addr learned my addr 172.21.15.88:0/3422793919 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:14:35.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:35.800+0000 7f36d4dbf700 1 -- 172.21.15.88:0/3422793919 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c4005040 con 0x7f36d0060200 2024-02-19T22:14:35.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:35.800+0000 7f36d4dbf700 1 --2- 172.21.15.88:0/3422793919 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f36d0060200 0x7f36d01089a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f36c40089f0 tx=0x7f36c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:35.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:35.800+0000 7f36cd7fa700 1 -- 172.21.15.88:0/3422793919 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f36c4009320 con 0x7f36d0060200 2024-02-19T22:14:35.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:35.800+0000 7f36d7023700 1 -- 172.21.15.88:0/3422793919 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36d0109090 con 0x7f36d0060200 2024-02-19T22:14:35.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:35.800+0000 7f36cd7fa700 1 -- 172.21.15.88:0/3422793919 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f36c4013380 con 0x7f36d0060200 2024-02-19T22:14:35.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:35.800+0000 7f36cd7fa700 1 -- 172.21.15.88:0/3422793919 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f36c4014e20 con 0x7f36d0060200 2024-02-19T22:14:35.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:35.800+0000 7f36d7023700 1 -- 172.21.15.88:0/3422793919 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36d0109520 con 0x7f36d0060200 2024-02-19T22:14:35.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:35.801+0000 7f36cd7fa700 1 -- 172.21.15.88:0/3422793919 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f36c40263a0 con 0x7f36d0060200 2024-02-19T22:14:35.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:35.801+0000 7f36cd7fa700 1 --2- 172.21.15.88:0/3422793919 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f36c002ff10 0x7f36c00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:35.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:35.801+0000 7f36cd7fa700 1 -- 172.21.15.88:0/3422793919 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f36c4040ea0 con 0x7f36d0060200 2024-02-19T22:14:35.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:35.801+0000 7f36d7023700 1 -- 172.21.15.88:0/3422793919 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36d0045ad0 con 0x7f36d0060200 2024-02-19T22:14:35.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:35.801+0000 7f36cffff700 1 --2- 172.21.15.88:0/3422793919 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f36c002ff10 0x7f36c00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:35.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:35.801+0000 7f36cffff700 1 --2- 172.21.15.88:0/3422793919 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f36c002ff10 0x7f36c00323d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f36bc0096a0 tx=0x7f36bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:35.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:35.806+0000 7f36cd7fa700 1 -- 172.21.15.88:0/3422793919 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f36c4012920 con 0x7f36d0060200 2024-02-19T22:14:36.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:36.011+0000 7f36d7023700 1 -- 172.21.15.88:0/3422793919 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f36d010a4e0 con 0x7f36d0060200 2024-02-19T22:14:36.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:36.012+0000 7f36cd7fa700 1 -- 172.21.15.88:0/3422793919 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f36c402b020 con 0x7f36d0060200 2024-02-19T22:14:36.014 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:14:36.015 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:14:36.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:36.014+0000 7f36d7023700 1 -- 172.21.15.88:0/3422793919 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f36c002ff10 msgr2=0x7f36c00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:36.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:36.014+0000 7f36d7023700 1 --2- 172.21.15.88:0/3422793919 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f36c002ff10 0x7f36c00323d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f36bc0096a0 tx=0x7f36bc006b40).stop 2024-02-19T22:14:36.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:36.014+0000 7f36d7023700 1 -- 172.21.15.88:0/3422793919 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f36d0060200 msgr2=0x7f36d01089a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:36.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:36.014+0000 7f36d7023700 1 --2- 172.21.15.88:0/3422793919 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f36d0060200 0x7f36d01089a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f36c40089f0 tx=0x7f36c400b040).stop 2024-02-19T22:14:36.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:36.014+0000 7f36d7023700 1 -- 172.21.15.88:0/3422793919 shutdown_connections 2024-02-19T22:14:36.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:36.014+0000 7f36d7023700 1 --2- 172.21.15.88:0/3422793919 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f36c002ff10 0x7f36c00323d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:36.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:36.014+0000 7f36d7023700 1 --2- 172.21.15.88:0/3422793919 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f36d0060200 0x7f36d01089a0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:36.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:36.014+0000 7f36d7023700 1 -- 172.21.15.88:0/3422793919 >> 172.21.15.88:0/3422793919 conn(0x7f36d00f0b70 msgr2=0x7f36d00f15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:36.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:36.014+0000 7f36d7023700 1 -- 172.21.15.88:0/3422793919 shutdown_connections 2024-02-19T22:14:36.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:36.014+0000 7f36d7023700 1 -- 172.21.15.88:0/3422793919 wait complete. 2024-02-19T22:14:36.017 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:14:36.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:36 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:14:36.013691+0000 mon.smithi052 (mon.0) 194 : audit [DBG] from='client.? 172.21.15.88:0/3422793919' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:37.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:37 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:14:35.661980+0000 mgr.smithi052.ghqrtk (mgr.14162) 147 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:37.609 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:37.609 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:39.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:39.425+0000 7f79db0de700 1 -- 172.21.15.88:0/2057675930 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f79d40f5410 msgr2=0x7f79d40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:39.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:39.425+0000 7f79db0de700 1 --2- 172.21.15.88:0/2057675930 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f79d40f5410 0x7f79d40f5830 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f79c8009fb0 tx=0x7f79c8009b30).stop 2024-02-19T22:14:39.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:39.425+0000 7f79db0de700 1 -- 172.21.15.88:0/2057675930 shutdown_connections 2024-02-19T22:14:39.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:39.425+0000 7f79db0de700 1 --2- 172.21.15.88:0/2057675930 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f79d40f5410 0x7f79d40f5830 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:39.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:39.425+0000 7f79db0de700 1 -- 172.21.15.88:0/2057675930 >> 172.21.15.88:0/2057675930 conn(0x7f79d40f0b70 msgr2=0x7f79d40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:39.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:39.425+0000 7f79db0de700 1 -- 172.21.15.88:0/2057675930 shutdown_connections 2024-02-19T22:14:39.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:39.425+0000 7f79db0de700 1 -- 172.21.15.88:0/2057675930 wait complete. 2024-02-19T22:14:39.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:39.426+0000 7f79db0de700 1 Processor -- start 2024-02-19T22:14:39.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:39.426+0000 7f79db0de700 1 -- start start 2024-02-19T22:14:39.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:39.426+0000 7f79db0de700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f79d40f5410 0x7f79d41094d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:39.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:39.426+0000 7f79db0de700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d4060280 con 0x7f79d40f5410 2024-02-19T22:14:39.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:39.426+0000 7f79d8e7a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f79d40f5410 0x7f79d41094d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:39.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:39.426+0000 7f79d8e7a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f79d40f5410 0x7f79d41094d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:37020/0 (socket says 172.21.15.88:37020) 2024-02-19T22:14:39.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:39.426+0000 7f79d8e7a700 1 -- 172.21.15.88:0/1560221614 learned_addr learned my addr 172.21.15.88:0/1560221614 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:14:39.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:39.427+0000 7f79d8e7a700 1 -- 172.21.15.88:0/1560221614 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79c8005040 con 0x7f79d40f5410 2024-02-19T22:14:39.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:39.427+0000 7f79d8e7a700 1 --2- 172.21.15.88:0/1560221614 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f79d40f5410 0x7f79d41094d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f79c8005010 tx=0x7f79c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:39.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:39.427+0000 7f79d17fa700 1 -- 172.21.15.88:0/1560221614 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f79c8009320 con 0x7f79d40f5410 2024-02-19T22:14:39.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:39.427+0000 7f79db0de700 1 -- 172.21.15.88:0/1560221614 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79d4060480 con 0x7f79d40f5410 2024-02-19T22:14:39.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:39.427+0000 7f79d17fa700 1 -- 172.21.15.88:0/1560221614 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f79c8013380 con 0x7f79d40f5410 2024-02-19T22:14:39.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:39.427+0000 7f79d17fa700 1 -- 172.21.15.88:0/1560221614 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f79c8014e50 con 0x7f79d40f5410 2024-02-19T22:14:39.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:39.427+0000 7f79db0de700 1 -- 172.21.15.88:0/1560221614 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79d4109c00 con 0x7f79d40f5410 2024-02-19T22:14:39.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:39.428+0000 7f79d17fa700 1 -- 172.21.15.88:0/1560221614 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f79c80095f0 con 0x7f79d40f5410 2024-02-19T22:14:39.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:39.428+0000 7f79db0de700 1 -- 172.21.15.88:0/1560221614 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79d4045ad0 con 0x7f79d40f5410 2024-02-19T22:14:39.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:39.428+0000 7f79d17fa700 1 --2- 172.21.15.88:0/1560221614 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f79c402ff10 0x7f79c40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:39.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:39.428+0000 7f79d17fa700 1 -- 172.21.15.88:0/1560221614 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f79c8040e90 con 0x7f79d40f5410 2024-02-19T22:14:39.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:39.428+0000 7f79d3fff700 1 --2- 172.21.15.88:0/1560221614 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f79c402ff10 0x7f79c40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:39.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:39.428+0000 7f79d3fff700 1 --2- 172.21.15.88:0/1560221614 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f79c402ff10 0x7f79c40323d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f79c00096a0 tx=0x7f79c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:39.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:39.433+0000 7f79d17fa700 1 -- 172.21.15.88:0/1560221614 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f79c8014420 con 0x7f79d40f5410 2024-02-19T22:14:39.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:14:37.662244+0000 mgr.smithi052.ghqrtk (mgr.14162) 148 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:39.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:39.638+0000 7f79db0de700 1 -- 172.21.15.88:0/1560221614 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f79d40fafa0 con 0x7f79d40f5410 2024-02-19T22:14:39.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:39.639+0000 7f79d17fa700 1 -- 172.21.15.88:0/1560221614 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f79c8010230 con 0x7f79d40f5410 2024-02-19T22:14:39.641 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:14:39.641 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:14:39.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:39.641+0000 7f79db0de700 1 -- 172.21.15.88:0/1560221614 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f79c402ff10 msgr2=0x7f79c40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:39.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:39.641+0000 7f79db0de700 1 --2- 172.21.15.88:0/1560221614 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f79c402ff10 0x7f79c40323d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f79c00096a0 tx=0x7f79c0006b40).stop 2024-02-19T22:14:39.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:39.641+0000 7f79db0de700 1 -- 172.21.15.88:0/1560221614 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f79d40f5410 msgr2=0x7f79d41094d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:39.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:39.641+0000 7f79db0de700 1 --2- 172.21.15.88:0/1560221614 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f79d40f5410 0x7f79d41094d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f79c8005010 tx=0x7f79c800b040).stop 2024-02-19T22:14:39.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:39.641+0000 7f79db0de700 1 -- 172.21.15.88:0/1560221614 shutdown_connections 2024-02-19T22:14:39.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:39.641+0000 7f79db0de700 1 --2- 172.21.15.88:0/1560221614 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f79c402ff10 0x7f79c40323d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:39.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:39.641+0000 7f79db0de700 1 --2- 172.21.15.88:0/1560221614 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f79d40f5410 0x7f79d41094d0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:39.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:39.641+0000 7f79db0de700 1 -- 172.21.15.88:0/1560221614 >> 172.21.15.88:0/1560221614 conn(0x7f79d40f0b70 msgr2=0x7f79d40f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:39.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:39.641+0000 7f79db0de700 1 -- 172.21.15.88:0/1560221614 shutdown_connections 2024-02-19T22:14:39.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:39.641+0000 7f79db0de700 1 -- 172.21.15.88:0/1560221614 wait complete. 2024-02-19T22:14:39.644 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:14:40.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:40 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:14:39.639382+0000 mon.smithi052 (mon.0) 195 : audit [DBG] from='client.? 172.21.15.88:0/1560221614' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:41.306 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:41.306 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:41.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:41 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:14:39.662547+0000 mgr.smithi052.ghqrtk (mgr.14162) 149 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:43.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:43.163+0000 7fa8bf7c9700 1 -- 172.21.15.88:0/429141508 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa8b80f5410 msgr2=0x7fa8b80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:43.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:43.163+0000 7fa8bf7c9700 1 --2- 172.21.15.88:0/429141508 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa8b80f5410 0x7fa8b80f5830 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7fa8a8009fb0 tx=0x7fa8a8009b30).stop 2024-02-19T22:14:43.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:43.163+0000 7fa8bf7c9700 1 -- 172.21.15.88:0/429141508 shutdown_connections 2024-02-19T22:14:43.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:43.163+0000 7fa8bf7c9700 1 --2- 172.21.15.88:0/429141508 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa8b80f5410 0x7fa8b80f5830 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:43.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:43.163+0000 7fa8bf7c9700 1 -- 172.21.15.88:0/429141508 >> 172.21.15.88:0/429141508 conn(0x7fa8b80f0b70 msgr2=0x7fa8b80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:43.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:43.164+0000 7fa8bf7c9700 1 -- 172.21.15.88:0/429141508 shutdown_connections 2024-02-19T22:14:43.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:43.164+0000 7fa8bf7c9700 1 -- 172.21.15.88:0/429141508 wait complete. 2024-02-19T22:14:43.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:43.164+0000 7fa8bf7c9700 1 Processor -- start 2024-02-19T22:14:43.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:43.164+0000 7fa8bf7c9700 1 -- start start 2024-02-19T22:14:43.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:43.164+0000 7fa8bf7c9700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa8b80f5410 0x7fa8b8109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:43.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:43.164+0000 7fa8bf7c9700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b8060220 con 0x7fa8b80f5410 2024-02-19T22:14:43.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:43.164+0000 7fa8bd565700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa8b80f5410 0x7fa8b8109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:43.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:43.164+0000 7fa8bd565700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa8b80f5410 0x7fa8b8109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:37050/0 (socket says 172.21.15.88:37050) 2024-02-19T22:14:43.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:43.164+0000 7fa8bd565700 1 -- 172.21.15.88:0/3428958531 learned_addr learned my addr 172.21.15.88:0/3428958531 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:14:43.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:43.165+0000 7fa8bd565700 1 -- 172.21.15.88:0/3428958531 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a8005040 con 0x7fa8b80f5410 2024-02-19T22:14:43.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:43.165+0000 7fa8bd565700 1 --2- 172.21.15.88:0/3428958531 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa8b80f5410 0x7fa8b8109500 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fa8a8005010 tx=0x7fa8a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:43.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:43.165+0000 7fa8adffb700 1 -- 172.21.15.88:0/3428958531 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8a8009320 con 0x7fa8b80f5410 2024-02-19T22:14:43.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:43.165+0000 7fa8bf7c9700 1 -- 172.21.15.88:0/3428958531 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8b8060420 con 0x7fa8b80f5410 2024-02-19T22:14:43.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:43.165+0000 7fa8adffb700 1 -- 172.21.15.88:0/3428958531 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa8a8013380 con 0x7fa8b80f5410 2024-02-19T22:14:43.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:43.165+0000 7fa8adffb700 1 -- 172.21.15.88:0/3428958531 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8a8014e50 con 0x7fa8b80f5410 2024-02-19T22:14:43.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:43.165+0000 7fa8bf7c9700 1 -- 172.21.15.88:0/3428958531 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8b8109bb0 con 0x7fa8b80f5410 2024-02-19T22:14:43.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:43.166+0000 7fa8adffb700 1 -- 172.21.15.88:0/3428958531 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa8a80095f0 con 0x7fa8b80f5410 2024-02-19T22:14:43.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:43.166+0000 7fa8bf7c9700 1 -- 172.21.15.88:0/3428958531 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8b8059100 con 0x7fa8b80f5410 2024-02-19T22:14:43.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:43.166+0000 7fa8adffb700 1 --2- 172.21.15.88:0/3428958531 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa8a402ff10 0x7fa8a40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:43.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:43.166+0000 7fa8adffb700 1 -- 172.21.15.88:0/3428958531 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa8a8040e70 con 0x7fa8b80f5410 2024-02-19T22:14:43.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:43.167+0000 7fa8bcd64700 1 --2- 172.21.15.88:0/3428958531 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa8a402ff10 0x7fa8a40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:43.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:43.167+0000 7fa8bcd64700 1 --2- 172.21.15.88:0/3428958531 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa8a402ff10 0x7fa8a40323d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fa8b40096a0 tx=0x7fa8b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:43.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:43.171+0000 7fa8adffb700 1 -- 172.21.15.88:0/3428958531 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa8a8014420 con 0x7fa8b80f5410 2024-02-19T22:14:43.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:43.377+0000 7fa8bf7c9700 1 -- 172.21.15.88:0/3428958531 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa8b8045ad0 con 0x7fa8b80f5410 2024-02-19T22:14:43.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:43.378+0000 7fa8adffb700 1 -- 172.21.15.88:0/3428958531 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa8a8010230 con 0x7fa8b80f5410 2024-02-19T22:14:43.379 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:14:43.379 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:14:43.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:43.379+0000 7fa8bf7c9700 1 -- 172.21.15.88:0/3428958531 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa8a402ff10 msgr2=0x7fa8a40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:43.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:43.379+0000 7fa8bf7c9700 1 --2- 172.21.15.88:0/3428958531 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa8a402ff10 0x7fa8a40323d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fa8b40096a0 tx=0x7fa8b4006b40).stop 2024-02-19T22:14:43.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:43.379+0000 7fa8bf7c9700 1 -- 172.21.15.88:0/3428958531 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa8b80f5410 msgr2=0x7fa8b8109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:43.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:43.379+0000 7fa8bf7c9700 1 --2- 172.21.15.88:0/3428958531 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa8b80f5410 0x7fa8b8109500 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fa8a8005010 tx=0x7fa8a800b040).stop 2024-02-19T22:14:43.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:43.380+0000 7fa8bf7c9700 1 -- 172.21.15.88:0/3428958531 shutdown_connections 2024-02-19T22:14:43.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:43.380+0000 7fa8bf7c9700 1 --2- 172.21.15.88:0/3428958531 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa8a402ff10 0x7fa8a40323d0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:43.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:43.380+0000 7fa8bf7c9700 1 --2- 172.21.15.88:0/3428958531 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa8b80f5410 0x7fa8b8109500 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:43.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:43.380+0000 7fa8bf7c9700 1 -- 172.21.15.88:0/3428958531 >> 172.21.15.88:0/3428958531 conn(0x7fa8b80f0b70 msgr2=0x7fa8b80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:43.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:43.380+0000 7fa8bf7c9700 1 -- 172.21.15.88:0/3428958531 shutdown_connections 2024-02-19T22:14:43.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:43.380+0000 7fa8bf7c9700 1 -- 172.21.15.88:0/3428958531 wait complete. 2024-02-19T22:14:43.381 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:14:43.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:43 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:14:41.662922+0000 mgr.smithi052.ghqrtk (mgr.14162) 150 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:44.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:44 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:14:43.378374+0000 mon.smithi052 (mon.0) 196 : audit [DBG] from='client.? 172.21.15.88:0/3428958531' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:44.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:44 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:14:43.749343+0000 mon.smithi052 (mon.0) 197 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi052.ghqrtk/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:14:44.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:44 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:14:43.752597+0000 mon.smithi052 (mon.0) 198 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi052.ghqrtk/trash_purge_schedule"}]: dispatch 2024-02-19T22:14:45.022 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:45.023 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:45.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:45 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:14:43.663330+0000 mgr.smithi052.ghqrtk (mgr.14162) 151 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:46.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:46.775+0000 7f3afbaa1700 1 -- 172.21.15.88:0/4042169455 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3af40f5410 msgr2=0x7f3af40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:46.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:46.775+0000 7f3afbaa1700 1 --2- 172.21.15.88:0/4042169455 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3af40f5410 0x7f3af40f5830 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f3ae4009fb0 tx=0x7f3ae4009b30).stop 2024-02-19T22:14:46.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:46.775+0000 7f3afbaa1700 1 -- 172.21.15.88:0/4042169455 shutdown_connections 2024-02-19T22:14:46.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:46.775+0000 7f3afbaa1700 1 --2- 172.21.15.88:0/4042169455 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3af40f5410 0x7f3af40f5830 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:46.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:46.775+0000 7f3afbaa1700 1 -- 172.21.15.88:0/4042169455 >> 172.21.15.88:0/4042169455 conn(0x7f3af40f0b70 msgr2=0x7f3af40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:46.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:46.775+0000 7f3afbaa1700 1 -- 172.21.15.88:0/4042169455 shutdown_connections 2024-02-19T22:14:46.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:46.775+0000 7f3afbaa1700 1 -- 172.21.15.88:0/4042169455 wait complete. 2024-02-19T22:14:46.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:46.776+0000 7f3afbaa1700 1 Processor -- start 2024-02-19T22:14:46.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:46.776+0000 7f3afbaa1700 1 -- start start 2024-02-19T22:14:46.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:46.776+0000 7f3afbaa1700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3af40f5410 0x7f3af41067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:46.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:46.776+0000 7f3afbaa1700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af4106cd0 con 0x7f3af40f5410 2024-02-19T22:14:46.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:46.776+0000 7f3af983d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3af40f5410 0x7f3af41067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:46.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:46.776+0000 7f3af983d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3af40f5410 0x7f3af41067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:39052/0 (socket says 172.21.15.88:39052) 2024-02-19T22:14:46.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:46.776+0000 7f3af983d700 1 -- 172.21.15.88:0/762531156 learned_addr learned my addr 172.21.15.88:0/762531156 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:14:46.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:46.776+0000 7f3af983d700 1 -- 172.21.15.88:0/762531156 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae4005040 con 0x7f3af40f5410 2024-02-19T22:14:46.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:46.776+0000 7f3af983d700 1 --2- 172.21.15.88:0/762531156 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3af40f5410 0x7f3af41067e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f3ae4005010 tx=0x7f3ae400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:46.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:46.777+0000 7f3aea7fc700 1 -- 172.21.15.88:0/762531156 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ae4009320 con 0x7f3af40f5410 2024-02-19T22:14:46.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:46.777+0000 7f3aea7fc700 1 -- 172.21.15.88:0/762531156 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3ae4013380 con 0x7f3af40f5410 2024-02-19T22:14:46.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:46.777+0000 7f3afbaa1700 1 -- 172.21.15.88:0/762531156 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3af4106ed0 con 0x7f3af40f5410 2024-02-19T22:14:46.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:46.777+0000 7f3aea7fc700 1 -- 172.21.15.88:0/762531156 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ae4014e50 con 0x7f3af40f5410 2024-02-19T22:14:46.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:46.777+0000 7f3afbaa1700 1 -- 172.21.15.88:0/762531156 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3af4107360 con 0x7f3af40f5410 2024-02-19T22:14:46.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:46.777+0000 7f3afbaa1700 1 -- 172.21.15.88:0/762531156 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3af4045ad0 con 0x7f3af40f5410 2024-02-19T22:14:46.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:46.777+0000 7f3aea7fc700 1 -- 172.21.15.88:0/762531156 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3ae40095f0 con 0x7f3af40f5410 2024-02-19T22:14:46.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:46.778+0000 7f3aea7fc700 1 --2- 172.21.15.88:0/762531156 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f3ae002ff10 0x7f3ae00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:46.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:46.778+0000 7f3aea7fc700 1 -- 172.21.15.88:0/762531156 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3ae4040fa0 con 0x7f3af40f5410 2024-02-19T22:14:46.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:46.778+0000 7f3af903c700 1 --2- 172.21.15.88:0/762531156 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f3ae002ff10 0x7f3ae00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:46.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:46.778+0000 7f3af903c700 1 --2- 172.21.15.88:0/762531156 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f3ae002ff10 0x7f3ae00323d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f3af00096a0 tx=0x7f3af0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:46.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:46.782+0000 7f3aea7fc700 1 -- 172.21.15.88:0/762531156 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3ae4014420 con 0x7f3af40f5410 2024-02-19T22:14:46.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:46.987+0000 7f3afbaa1700 1 -- 172.21.15.88:0/762531156 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3af4108210 con 0x7f3af40f5410 2024-02-19T22:14:46.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:46.987+0000 7f3aea7fc700 1 -- 172.21.15.88:0/762531156 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3ae4010230 con 0x7f3af40f5410 2024-02-19T22:14:46.989 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:14:46.989 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:14:46.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:46.989+0000 7f3afbaa1700 1 -- 172.21.15.88:0/762531156 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f3ae002ff10 msgr2=0x7f3ae00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:46.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:46.989+0000 7f3afbaa1700 1 --2- 172.21.15.88:0/762531156 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f3ae002ff10 0x7f3ae00323d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f3af00096a0 tx=0x7f3af0006b40).stop 2024-02-19T22:14:46.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:46.989+0000 7f3afbaa1700 1 -- 172.21.15.88:0/762531156 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3af40f5410 msgr2=0x7f3af41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:46.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:46.989+0000 7f3afbaa1700 1 --2- 172.21.15.88:0/762531156 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3af40f5410 0x7f3af41067e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f3ae4005010 tx=0x7f3ae400b040).stop 2024-02-19T22:14:46.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:46.989+0000 7f3afbaa1700 1 -- 172.21.15.88:0/762531156 shutdown_connections 2024-02-19T22:14:46.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:46.989+0000 7f3afbaa1700 1 --2- 172.21.15.88:0/762531156 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f3ae002ff10 0x7f3ae00323d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:46.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:46.989+0000 7f3afbaa1700 1 --2- 172.21.15.88:0/762531156 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3af40f5410 0x7f3af41067e0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:46.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:46.989+0000 7f3afbaa1700 1 -- 172.21.15.88:0/762531156 >> 172.21.15.88:0/762531156 conn(0x7f3af40f0b70 msgr2=0x7f3af40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:46.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:46.989+0000 7f3afbaa1700 1 -- 172.21.15.88:0/762531156 shutdown_connections 2024-02-19T22:14:46.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:46.989+0000 7f3afbaa1700 1 -- 172.21.15.88:0/762531156 wait complete. 2024-02-19T22:14:46.991 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:14:47.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:47 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:14:45.663678+0000 mgr.smithi052.ghqrtk (mgr.14162) 152 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:47.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:47 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:14:46.987973+0000 mon.smithi052 (mon.0) 199 : audit [DBG] from='client.? 172.21.15.88:0/762531156' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:48.658 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:48.659 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:49.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:49 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:14:47.664048+0000 mgr.smithi052.ghqrtk (mgr.14162) 153 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:50.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:50.419+0000 7f0f30d72700 1 -- 172.21.15.88:0/2614614961 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f2c060220 msgr2=0x7f0f2c0f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:50.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:50.419+0000 7f0f30d72700 1 --2- 172.21.15.88:0/2614614961 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f2c060220 0x7f0f2c0f9150 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f0f1c009fb0 tx=0x7f0f1c009b30).stop 2024-02-19T22:14:50.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:50.419+0000 7f0f30d72700 1 -- 172.21.15.88:0/2614614961 shutdown_connections 2024-02-19T22:14:50.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:50.419+0000 7f0f30d72700 1 --2- 172.21.15.88:0/2614614961 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f2c060220 0x7f0f2c0f9150 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:50.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:50.419+0000 7f0f30d72700 1 -- 172.21.15.88:0/2614614961 >> 172.21.15.88:0/2614614961 conn(0x7f0f2c0f0b70 msgr2=0x7f0f2c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:50.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:50.419+0000 7f0f30d72700 1 -- 172.21.15.88:0/2614614961 shutdown_connections 2024-02-19T22:14:50.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:50.419+0000 7f0f30d72700 1 -- 172.21.15.88:0/2614614961 wait complete. 2024-02-19T22:14:50.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:50.420+0000 7f0f30d72700 1 Processor -- start 2024-02-19T22:14:50.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:50.420+0000 7f0f30d72700 1 -- start start 2024-02-19T22:14:50.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:50.420+0000 7f0f30d72700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f2c060220 0x7f0f2c1067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:50.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:50.420+0000 7f0f30d72700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2c106c90 con 0x7f0f2c060220 2024-02-19T22:14:50.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:50.420+0000 7f0f2a59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f2c060220 0x7f0f2c1067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:50.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:50.420+0000 7f0f2a59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f2c060220 0x7f0f2c1067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:39074/0 (socket says 172.21.15.88:39074) 2024-02-19T22:14:50.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:50.420+0000 7f0f2a59c700 1 -- 172.21.15.88:0/3100718847 learned_addr learned my addr 172.21.15.88:0/3100718847 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:14:50.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:50.421+0000 7f0f2a59c700 1 -- 172.21.15.88:0/3100718847 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f1c005040 con 0x7f0f2c060220 2024-02-19T22:14:50.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:50.421+0000 7f0f2a59c700 1 --2- 172.21.15.88:0/3100718847 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f2c060220 0x7f0f2c1067a0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f0f1c0089f0 tx=0x7f0f1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:50.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:50.421+0000 7f0f22ffd700 1 -- 172.21.15.88:0/3100718847 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f1c009320 con 0x7f0f2c060220 2024-02-19T22:14:50.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:50.421+0000 7f0f30d72700 1 -- 172.21.15.88:0/3100718847 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f2c106e90 con 0x7f0f2c060220 2024-02-19T22:14:50.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:50.421+0000 7f0f22ffd700 1 -- 172.21.15.88:0/3100718847 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0f1c013380 con 0x7f0f2c060220 2024-02-19T22:14:50.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:50.421+0000 7f0f22ffd700 1 -- 172.21.15.88:0/3100718847 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f1c014e20 con 0x7f0f2c060220 2024-02-19T22:14:50.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:50.421+0000 7f0f30d72700 1 -- 172.21.15.88:0/3100718847 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f2c107320 con 0x7f0f2c060220 2024-02-19T22:14:50.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:50.422+0000 7f0f22ffd700 1 -- 172.21.15.88:0/3100718847 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0f1c0095f0 con 0x7f0f2c060220 2024-02-19T22:14:50.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:50.422+0000 7f0f30d72700 1 -- 172.21.15.88:0/3100718847 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f2c045ad0 con 0x7f0f2c060220 2024-02-19T22:14:50.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:50.422+0000 7f0f22ffd700 1 --2- 172.21.15.88:0/3100718847 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f0f1802ff10 0x7f0f180323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:50.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:50.422+0000 7f0f22ffd700 1 -- 172.21.15.88:0/3100718847 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0f1c040ea0 con 0x7f0f2c060220 2024-02-19T22:14:50.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:50.422+0000 7f0f29d9b700 1 --2- 172.21.15.88:0/3100718847 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f0f1802ff10 0x7f0f180323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:50.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:50.422+0000 7f0f29d9b700 1 --2- 172.21.15.88:0/3100718847 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f0f1802ff10 0x7f0f180323d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f0f140096a0 tx=0x7f0f14006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:50.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:50.427+0000 7f0f22ffd700 1 -- 172.21.15.88:0/3100718847 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0f1c0268c0 con 0x7f0f2c060220 2024-02-19T22:14:50.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:50.632+0000 7f0f30d72700 1 -- 172.21.15.88:0/3100718847 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0f2c1081d0 con 0x7f0f2c060220 2024-02-19T22:14:50.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:50.632+0000 7f0f22ffd700 1 -- 172.21.15.88:0/3100718847 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0f1c01d980 con 0x7f0f2c060220 2024-02-19T22:14:50.634 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:14:50.634 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:14:50.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:50.634+0000 7f0f30d72700 1 -- 172.21.15.88:0/3100718847 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f0f1802ff10 msgr2=0x7f0f180323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:50.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:50.634+0000 7f0f30d72700 1 --2- 172.21.15.88:0/3100718847 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f0f1802ff10 0x7f0f180323d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f0f140096a0 tx=0x7f0f14006b40).stop 2024-02-19T22:14:50.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:50.634+0000 7f0f30d72700 1 -- 172.21.15.88:0/3100718847 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f2c060220 msgr2=0x7f0f2c1067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:50.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:50.634+0000 7f0f30d72700 1 --2- 172.21.15.88:0/3100718847 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f2c060220 0x7f0f2c1067a0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f0f1c0089f0 tx=0x7f0f1c00b040).stop 2024-02-19T22:14:50.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:50.634+0000 7f0f30d72700 1 -- 172.21.15.88:0/3100718847 shutdown_connections 2024-02-19T22:14:50.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:50.634+0000 7f0f30d72700 1 --2- 172.21.15.88:0/3100718847 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f0f1802ff10 0x7f0f180323d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:50.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:50.634+0000 7f0f30d72700 1 --2- 172.21.15.88:0/3100718847 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f2c060220 0x7f0f2c1067a0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:50.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:50.634+0000 7f0f30d72700 1 -- 172.21.15.88:0/3100718847 >> 172.21.15.88:0/3100718847 conn(0x7f0f2c0f0b70 msgr2=0x7f0f2c0f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:50.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:50.634+0000 7f0f30d72700 1 -- 172.21.15.88:0/3100718847 shutdown_connections 2024-02-19T22:14:50.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:50.634+0000 7f0f30d72700 1 -- 172.21.15.88:0/3100718847 wait complete. 2024-02-19T22:14:50.638 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:14:51.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:51 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:14:49.664399+0000 mgr.smithi052.ghqrtk (mgr.14162) 154 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:51.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:51 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:14:50.632830+0000 mon.smithi052 (mon.0) 200 : audit [DBG] from='client.? 172.21.15.88:0/3100718847' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:52.269 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:52.270 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:53.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:53 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:14:51.664741+0000 mgr.smithi052.ghqrtk (mgr.14162) 155 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:53.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:53.980+0000 7f7178d4b700 1 -- 172.21.15.88:0/3540552184 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f71740f6510 msgr2=0x7f71740f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:53.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:53.980+0000 7f7178d4b700 1 --2- 172.21.15.88:0/3540552184 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f71740f6510 0x7f71740f6930 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f715c009fb0 tx=0x7f715c009b30).stop 2024-02-19T22:14:53.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:53.980+0000 7f7178d4b700 1 -- 172.21.15.88:0/3540552184 shutdown_connections 2024-02-19T22:14:53.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:53.980+0000 7f7178d4b700 1 --2- 172.21.15.88:0/3540552184 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f71740f6510 0x7f71740f6930 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:53.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:53.980+0000 7f7178d4b700 1 -- 172.21.15.88:0/3540552184 >> 172.21.15.88:0/3540552184 conn(0x7f71740f1c70 msgr2=0x7f71740f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:53.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:53.980+0000 7f7178d4b700 1 -- 172.21.15.88:0/3540552184 shutdown_connections 2024-02-19T22:14:53.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:53.980+0000 7f7178d4b700 1 -- 172.21.15.88:0/3540552184 wait complete. 2024-02-19T22:14:53.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:53.981+0000 7f7178d4b700 1 Processor -- start 2024-02-19T22:14:53.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:53.981+0000 7f7178d4b700 1 -- start start 2024-02-19T22:14:53.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:53.981+0000 7f7178d4b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f71740f6510 0x7f71741067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:53.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:53.981+0000 7f7178d4b700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7174106cd0 con 0x7f71740f6510 2024-02-19T22:14:53.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:53.981+0000 7f717259c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f71740f6510 0x7f71741067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:53.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:53.981+0000 7f717259c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f71740f6510 0x7f71741067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:54478/0 (socket says 172.21.15.88:54478) 2024-02-19T22:14:53.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:53.981+0000 7f717259c700 1 -- 172.21.15.88:0/490566704 learned_addr learned my addr 172.21.15.88:0/490566704 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:14:53.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:53.981+0000 7f717259c700 1 -- 172.21.15.88:0/490566704 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f715c005040 con 0x7f71740f6510 2024-02-19T22:14:53.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:53.982+0000 7f717259c700 1 --2- 172.21.15.88:0/490566704 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f71740f6510 0x7f71741067e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f715c005010 tx=0x7f715c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:53.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:53.982+0000 7f716affd700 1 -- 172.21.15.88:0/490566704 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f715c009320 con 0x7f71740f6510 2024-02-19T22:14:53.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:53.982+0000 7f7178d4b700 1 -- 172.21.15.88:0/490566704 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7174106ed0 con 0x7f71740f6510 2024-02-19T22:14:53.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:53.982+0000 7f716affd700 1 -- 172.21.15.88:0/490566704 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f715c013380 con 0x7f71740f6510 2024-02-19T22:14:53.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:53.982+0000 7f716affd700 1 -- 172.21.15.88:0/490566704 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f715c014e50 con 0x7f71740f6510 2024-02-19T22:14:53.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:53.982+0000 7f7178d4b700 1 -- 172.21.15.88:0/490566704 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7174107360 con 0x7f71740f6510 2024-02-19T22:14:53.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:53.983+0000 7f716affd700 1 -- 172.21.15.88:0/490566704 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f715c0095f0 con 0x7f71740f6510 2024-02-19T22:14:53.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:53.983+0000 7f7178d4b700 1 -- 172.21.15.88:0/490566704 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7174045ad0 con 0x7f71740f6510 2024-02-19T22:14:53.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:53.983+0000 7f716affd700 1 --2- 172.21.15.88:0/490566704 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f716002ff10 0x7f71600323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:53.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:53.983+0000 7f716affd700 1 -- 172.21.15.88:0/490566704 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f715c040e90 con 0x7f71740f6510 2024-02-19T22:14:53.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:53.983+0000 7f7171d9b700 1 --2- 172.21.15.88:0/490566704 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f716002ff10 0x7f71600323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:53.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:53.983+0000 7f7171d9b700 1 --2- 172.21.15.88:0/490566704 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f716002ff10 0x7f71600323d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f71640096a0 tx=0x7f7164006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:53.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:53.988+0000 7f716affd700 1 -- 172.21.15.88:0/490566704 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f715c014420 con 0x7f71740f6510 2024-02-19T22:14:54.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:54.196+0000 7f7178d4b700 1 -- 172.21.15.88:0/490566704 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7174108210 con 0x7f71740f6510 2024-02-19T22:14:54.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:54.196+0000 7f716affd700 1 -- 172.21.15.88:0/490566704 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f715c010230 con 0x7f71740f6510 2024-02-19T22:14:54.199 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:14:54.199 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:14:54.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:54.198+0000 7f7178d4b700 1 -- 172.21.15.88:0/490566704 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f716002ff10 msgr2=0x7f71600323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:54.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:54.198+0000 7f7178d4b700 1 --2- 172.21.15.88:0/490566704 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f716002ff10 0x7f71600323d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f71640096a0 tx=0x7f7164006b40).stop 2024-02-19T22:14:54.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:54.198+0000 7f7178d4b700 1 -- 172.21.15.88:0/490566704 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f71740f6510 msgr2=0x7f71741067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:54.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:54.198+0000 7f7178d4b700 1 --2- 172.21.15.88:0/490566704 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f71740f6510 0x7f71741067e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f715c005010 tx=0x7f715c00b040).stop 2024-02-19T22:14:54.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:54.198+0000 7f7178d4b700 1 -- 172.21.15.88:0/490566704 shutdown_connections 2024-02-19T22:14:54.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:54.198+0000 7f7178d4b700 1 --2- 172.21.15.88:0/490566704 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f716002ff10 0x7f71600323d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:54.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:54.198+0000 7f7178d4b700 1 --2- 172.21.15.88:0/490566704 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f71740f6510 0x7f71741067e0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:54.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:54.198+0000 7f7178d4b700 1 -- 172.21.15.88:0/490566704 >> 172.21.15.88:0/490566704 conn(0x7f71740f1c70 msgr2=0x7f71740f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:54.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:54.198+0000 7f7178d4b700 1 -- 172.21.15.88:0/490566704 shutdown_connections 2024-02-19T22:14:54.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:54.198+0000 7f7178d4b700 1 -- 172.21.15.88:0/490566704 wait complete. 2024-02-19T22:14:54.201 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:14:55.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:55 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:14:53.665090+0000 mgr.smithi052.ghqrtk (mgr.14162) 156 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:55.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:55 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:14:54.197788+0000 mon.smithi052 (mon.0) 201 : audit [DBG] from='client.? 172.21.15.88:0/490566704' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:55.835 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:55.836 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:56.343 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cephadm 2024-02-19T22:14:55.196302+0000 mgr.smithi052.ghqrtk (mgr.14162) 157 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi052 ... 2024-02-19T22:14:56.344 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi052 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-02-19T22:14:56.344 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-19T22:14:56.344 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: stat: stderr time="2024-02-19T22:10:33Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:14:56.344 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: stat: stderr time="2024-02-19T22:11:11Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:14:56.344 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: stat: stderr time="2024-02-19T22:11:48Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:14:56.344 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-19T22:14:56.344 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi052 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-02-19T22:14:56.344 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-19T22:14:56.344 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: stat: stderr time="2024-02-19T22:13:02Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:14:56.344 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: stat: stderr time="2024-02-19T22:13:40Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:14:56.344 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: stat: stderr time="2024-02-19T22:14:17Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:14:56.344 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-19T22:14:56.345 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: Traceback (most recent call last): 2024-02-19T22:14:56.345 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: File "/var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-19T22:14:56.345 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: main() 2024-02-19T22:14:56.345 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: File "/var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-19T22:14:56.345 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: r = ctx.func(ctx) 2024-02-19T22:14:56.345 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: File "/var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-19T22:14:56.345 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: return func(ctx) 2024-02-19T22:14:56.345 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: File "/var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-19T22:14:56.345 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-19T22:14:56.345 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: File "/var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-02-19T22:14:56.345 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-02-19T22:14:56.345 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: File "/var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-19T22:14:56.345 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: raise RuntimeError('uid/gid not found') 2024-02-19T22:14:56.345 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: RuntimeError: uid/gid not found 2024-02-19T22:14:56.346 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: Traceback (most recent call last): 2024-02-19T22:14:56.346 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-02-19T22:14:56.346 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: yield (conn, connr) 2024-02-19T22:14:56.346 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-02-19T22:14:56.346 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: code, '\n'.join(err))) 2024-02-19T22:14:56.346 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi052 ... 2024-02-19T22:14:56.346 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi052 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-02-19T22:14:56.346 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-19T22:14:56.346 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: stat: stderr time="2024-02-19T22:10:33Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:14:56.346 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: stat: stderr time="2024-02-19T22:11:11Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:14:56.346 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: stat: stderr time="2024-02-19T22:11:48Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:14:56.346 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-19T22:14:56.346 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi052 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-02-19T22:14:56.347 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-19T22:14:56.347 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: stat: stderr time="2024-02-19T22:13:02Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:14:56.347 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: stat: stderr time="2024-02-19T22:13:40Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:14:56.347 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: stat: stderr time="2024-02-19T22:14:17Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:14:56.347 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-19T22:14:56.347 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: Traceback (most recent call last): 2024-02-19T22:14:56.347 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: File "/var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-19T22:14:56.347 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: main() 2024-02-19T22:14:56.347 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: File "/var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-19T22:14:56.347 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: r = ctx.func(ctx) 2024-02-19T22:14:56.347 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: File "/var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-19T22:14:56.347 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: return func(ctx) 2024-02-19T22:14:56.347 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: File "/var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-19T22:14:56.347 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-19T22:14:56.347 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: File "/var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-02-19T22:14:56.348 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-02-19T22:14:56.348 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: File "/var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-19T22:14:56.348 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: raise RuntimeError('uid/gid not found') 2024-02-19T22:14:56.348 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: RuntimeError: uid/gid not found 2024-02-19T22:14:56.348 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:14:55.197517+0000 mon.smithi052 (mon.0) 202 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi052", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-19T22:14:56.348 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:14:55.199936+0000 mon.smithi052 (mon.0) 203 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi052", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-19T22:14:56.348 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:14:55.200738+0000 mon.smithi052 (mon.0) 204 : audit [DBG] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T22:14:56.348 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cephadm 2024-02-19T22:14:55.201467+0000 mgr.smithi052.ghqrtk (mgr.14162) 158 : cephadm [INF] Deploying daemon crash.smithi052 on smithi052 2024-02-19T22:14:57.532 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:57 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:14:55.665328+0000 mgr.smithi052.ghqrtk (mgr.14162) 159 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:57.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:57.677+0000 7f1206259700 1 -- 172.21.15.88:0/3322369136 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f12000f62a0 msgr2=0x7f12000f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:57.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:57.677+0000 7f1206259700 1 --2- 172.21.15.88:0/3322369136 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f12000f62a0 0x7f12000f66c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f11f0009fb0 tx=0x7f11f0009b30).stop 2024-02-19T22:14:57.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:57.677+0000 7f1206259700 1 -- 172.21.15.88:0/3322369136 shutdown_connections 2024-02-19T22:14:57.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:57.677+0000 7f1206259700 1 --2- 172.21.15.88:0/3322369136 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f12000f62a0 0x7f12000f66c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:57.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:57.677+0000 7f1206259700 1 -- 172.21.15.88:0/3322369136 >> 172.21.15.88:0/3322369136 conn(0x7f12000f1aa0 msgr2=0x7f12000f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:57.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:57.677+0000 7f1206259700 1 -- 172.21.15.88:0/3322369136 shutdown_connections 2024-02-19T22:14:57.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:57.677+0000 7f1206259700 1 -- 172.21.15.88:0/3322369136 wait complete. 2024-02-19T22:14:57.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:57.677+0000 7f1206259700 1 Processor -- start 2024-02-19T22:14:57.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:57.677+0000 7f1206259700 1 -- start start 2024-02-19T22:14:57.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:57.677+0000 7f1206259700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f12000f62a0 0x7f120006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:57.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:57.677+0000 7f1206259700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1200069e20 con 0x7f12000f62a0 2024-02-19T22:14:57.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:57.678+0000 7f1205257700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f12000f62a0 0x7f120006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:57.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:57.678+0000 7f1205257700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f12000f62a0 0x7f120006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:54492/0 (socket says 172.21.15.88:54492) 2024-02-19T22:14:57.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:57.678+0000 7f1205257700 1 -- 172.21.15.88:0/1699592020 learned_addr learned my addr 172.21.15.88:0/1699592020 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:14:57.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:57.678+0000 7f1205257700 1 -- 172.21.15.88:0/1699592020 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f0005040 con 0x7f12000f62a0 2024-02-19T22:14:57.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:57.678+0000 7f1205257700 1 --2- 172.21.15.88:0/1699592020 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f12000f62a0 0x7f120006b9a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f11f00089f0 tx=0x7f11f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:57.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:57.678+0000 7f11f5ffb700 1 -- 172.21.15.88:0/1699592020 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11f0009320 con 0x7f12000f62a0 2024-02-19T22:14:57.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:57.678+0000 7f11f5ffb700 1 -- 172.21.15.88:0/1699592020 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f11f0013380 con 0x7f12000f62a0 2024-02-19T22:14:57.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:57.678+0000 7f1206259700 1 -- 172.21.15.88:0/1699592020 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1200069fc0 con 0x7f12000f62a0 2024-02-19T22:14:57.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:57.678+0000 7f11f5ffb700 1 -- 172.21.15.88:0/1699592020 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11f0014e20 con 0x7f12000f62a0 2024-02-19T22:14:57.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:57.678+0000 7f1206259700 1 -- 172.21.15.88:0/1699592020 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f120006a450 con 0x7f12000f62a0 2024-02-19T22:14:57.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:57.679+0000 7f11f5ffb700 1 -- 172.21.15.88:0/1699592020 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f11f0014420 con 0x7f12000f62a0 2024-02-19T22:14:57.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:57.679+0000 7f1206259700 1 -- 172.21.15.88:0/1699592020 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1200059100 con 0x7f12000f62a0 2024-02-19T22:14:57.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:57.679+0000 7f11f5ffb700 1 --2- 172.21.15.88:0/1699592020 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f11ec02ff10 0x7f11ec0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:57.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:57.679+0000 7f11f5ffb700 1 -- 172.21.15.88:0/1699592020 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f11f0040ff0 con 0x7f12000f62a0 2024-02-19T22:14:57.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:57.679+0000 7f1204a56700 1 --2- 172.21.15.88:0/1699592020 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f11ec02ff10 0x7f11ec0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:57.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:57.680+0000 7f1204a56700 1 --2- 172.21.15.88:0/1699592020 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f11ec02ff10 0x7f11ec0323d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f11fc0096a0 tx=0x7f11fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:57.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:57.684+0000 7f11f5ffb700 1 -- 172.21.15.88:0/1699592020 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f11f00095f0 con 0x7f12000f62a0 2024-02-19T22:14:57.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:57.888+0000 7f1206259700 1 -- 172.21.15.88:0/1699592020 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f12000fbe30 con 0x7f12000f62a0 2024-02-19T22:14:57.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:57.889+0000 7f11f5ffb700 1 -- 172.21.15.88:0/1699592020 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f11f001db70 con 0x7f12000f62a0 2024-02-19T22:14:57.891 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:14:57.892 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:14:57.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:57.891+0000 7f1206259700 1 -- 172.21.15.88:0/1699592020 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f11ec02ff10 msgr2=0x7f11ec0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:57.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:57.891+0000 7f1206259700 1 --2- 172.21.15.88:0/1699592020 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f11ec02ff10 0x7f11ec0323d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f11fc0096a0 tx=0x7f11fc006b40).stop 2024-02-19T22:14:57.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:57.891+0000 7f1206259700 1 -- 172.21.15.88:0/1699592020 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f12000f62a0 msgr2=0x7f120006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:57.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:57.891+0000 7f1206259700 1 --2- 172.21.15.88:0/1699592020 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f12000f62a0 0x7f120006b9a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f11f00089f0 tx=0x7f11f000b040).stop 2024-02-19T22:14:57.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:57.891+0000 7f1206259700 1 -- 172.21.15.88:0/1699592020 shutdown_connections 2024-02-19T22:14:57.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:57.891+0000 7f1206259700 1 --2- 172.21.15.88:0/1699592020 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f11ec02ff10 0x7f11ec0323d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:57.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:57.891+0000 7f1206259700 1 --2- 172.21.15.88:0/1699592020 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f12000f62a0 0x7f120006b9a0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:57.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:57.891+0000 7f1206259700 1 -- 172.21.15.88:0/1699592020 >> 172.21.15.88:0/1699592020 conn(0x7f12000f1aa0 msgr2=0x7f12000f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:57.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:57.891+0000 7f1206259700 1 -- 172.21.15.88:0/1699592020 shutdown_connections 2024-02-19T22:14:57.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:14:57.891+0000 7f1206259700 1 -- 172.21.15.88:0/1699592020 wait complete. 2024-02-19T22:14:57.894 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:14:58.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:58 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:14:57.890558+0000 mon.smithi052 (mon.0) 205 : audit [DBG] from='client.? 172.21.15.88:0/1699592020' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:58.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:58 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:14:57.920434+0000 mon.smithi052 (mon.0) 206 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' 2024-02-19T22:14:58.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:58 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:14:58.076729+0000 mon.smithi052 (mon.0) 207 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' 2024-02-19T22:14:58.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:58 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:14:58.078847+0000 mon.smithi052 (mon.0) 208 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' 2024-02-19T22:14:58.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:58 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:14:58.079312+0000 mon.smithi052 (mon.0) 209 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-19T22:14:58.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:58 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:14:58.082115+0000 mon.smithi052 (mon.0) 210 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' 2024-02-19T22:14:59.421 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:59.421 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:59.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:59 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:14:57.665688+0000 mgr.smithi052.ghqrtk (mgr.14162) 160 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:59.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:59 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:14:58.079760+0000 mgr.smithi052.ghqrtk (mgr.14162) 161 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-19T22:14:59.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:14:59 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cephadm 2024-02-19T22:14:58.085120+0000 mgr.smithi052.ghqrtk (mgr.14162) 162 : cephadm [INF] Deploying daemon grafana.smithi052 on smithi052 2024-02-19T22:15:01.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:01.263+0000 7f7961a6e700 1 -- 172.21.15.88:0/1364497657 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f795c0f5410 msgr2=0x7f795c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:01.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:01.263+0000 7f7961a6e700 1 --2- 172.21.15.88:0/1364497657 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f795c0f5410 0x7f795c0f5830 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f7944009fb0 tx=0x7f7944009b30).stop 2024-02-19T22:15:01.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:01.263+0000 7f7961a6e700 1 -- 172.21.15.88:0/1364497657 shutdown_connections 2024-02-19T22:15:01.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:01.263+0000 7f7961a6e700 1 --2- 172.21.15.88:0/1364497657 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f795c0f5410 0x7f795c0f5830 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:01.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:01.263+0000 7f7961a6e700 1 -- 172.21.15.88:0/1364497657 >> 172.21.15.88:0/1364497657 conn(0x7f795c0f0b70 msgr2=0x7f795c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:01.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:01.263+0000 7f7961a6e700 1 -- 172.21.15.88:0/1364497657 shutdown_connections 2024-02-19T22:15:01.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:01.263+0000 7f7961a6e700 1 -- 172.21.15.88:0/1364497657 wait complete. 2024-02-19T22:15:01.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:01.264+0000 7f7961a6e700 1 Processor -- start 2024-02-19T22:15:01.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:01.264+0000 7f7961a6e700 1 -- start start 2024-02-19T22:15:01.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:01.264+0000 7f7961a6e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f795c0f5410 0x7f795c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:01.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:01.264+0000 7f7961a6e700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f795c106cd0 con 0x7f795c0f5410 2024-02-19T22:15:01.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:01.264+0000 7f795b7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f795c0f5410 0x7f795c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:01.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:01.264+0000 7f795b7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f795c0f5410 0x7f795c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:54522/0 (socket says 172.21.15.88:54522) 2024-02-19T22:15:01.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:01.264+0000 7f795b7fe700 1 -- 172.21.15.88:0/372327333 learned_addr learned my addr 172.21.15.88:0/372327333 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:15:01.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:01.264+0000 7f795b7fe700 1 -- 172.21.15.88:0/372327333 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7944005040 con 0x7f795c0f5410 2024-02-19T22:15:01.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:01.264+0000 7f795b7fe700 1 --2- 172.21.15.88:0/372327333 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f795c0f5410 0x7f795c1067e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f7944005010 tx=0x7f794400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:01.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:01.265+0000 7f7953fff700 1 -- 172.21.15.88:0/372327333 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7944009320 con 0x7f795c0f5410 2024-02-19T22:15:01.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:01.265+0000 7f7961a6e700 1 -- 172.21.15.88:0/372327333 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f795c106ed0 con 0x7f795c0f5410 2024-02-19T22:15:01.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:01.265+0000 7f7953fff700 1 -- 172.21.15.88:0/372327333 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7944013380 con 0x7f795c0f5410 2024-02-19T22:15:01.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:01.265+0000 7f7953fff700 1 -- 172.21.15.88:0/372327333 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7944014e50 con 0x7f795c0f5410 2024-02-19T22:15:01.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:01.265+0000 7f7961a6e700 1 -- 172.21.15.88:0/372327333 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f795c107360 con 0x7f795c0f5410 2024-02-19T22:15:01.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:01.265+0000 7f7953fff700 1 -- 172.21.15.88:0/372327333 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f79440095f0 con 0x7f795c0f5410 2024-02-19T22:15:01.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:01.266+0000 7f7961a6e700 1 -- 172.21.15.88:0/372327333 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f795c045ad0 con 0x7f795c0f5410 2024-02-19T22:15:01.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:01.266+0000 7f7953fff700 1 --2- 172.21.15.88:0/372327333 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f794802ff10 0x7f79480323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:01.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:01.266+0000 7f7953fff700 1 -- 172.21.15.88:0/372327333 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7944040e60 con 0x7f795c0f5410 2024-02-19T22:15:01.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:01.266+0000 7f795affd700 1 --2- 172.21.15.88:0/372327333 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f794802ff10 0x7f79480323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:01.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:01.266+0000 7f795affd700 1 --2- 172.21.15.88:0/372327333 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f794802ff10 0x7f79480323d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f794c0096a0 tx=0x7f794c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:01.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:01.271+0000 7f7953fff700 1 -- 172.21.15.88:0/372327333 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7944014420 con 0x7f795c0f5410 2024-02-19T22:15:01.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:01.474+0000 7f7961a6e700 1 -- 172.21.15.88:0/372327333 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f795c1081b0 con 0x7f795c0f5410 2024-02-19T22:15:01.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:01.475+0000 7f7953fff700 1 -- 172.21.15.88:0/372327333 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7944010230 con 0x7f795c0f5410 2024-02-19T22:15:01.477 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:15:01.478 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:01.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:01.477+0000 7f7961a6e700 1 -- 172.21.15.88:0/372327333 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f794802ff10 msgr2=0x7f79480323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:01.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:01.477+0000 7f7961a6e700 1 --2- 172.21.15.88:0/372327333 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f794802ff10 0x7f79480323d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f794c0096a0 tx=0x7f794c006b40).stop 2024-02-19T22:15:01.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:01.477+0000 7f7961a6e700 1 -- 172.21.15.88:0/372327333 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f795c0f5410 msgr2=0x7f795c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:01.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:01.477+0000 7f7961a6e700 1 --2- 172.21.15.88:0/372327333 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f795c0f5410 0x7f795c1067e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f7944005010 tx=0x7f794400b040).stop 2024-02-19T22:15:01.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:01.477+0000 7f7961a6e700 1 -- 172.21.15.88:0/372327333 shutdown_connections 2024-02-19T22:15:01.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:01.477+0000 7f7961a6e700 1 --2- 172.21.15.88:0/372327333 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f794802ff10 0x7f79480323d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:01.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:01.477+0000 7f7961a6e700 1 --2- 172.21.15.88:0/372327333 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f795c0f5410 0x7f795c1067e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:01.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:01.477+0000 7f7961a6e700 1 -- 172.21.15.88:0/372327333 >> 172.21.15.88:0/372327333 conn(0x7f795c0f0b70 msgr2=0x7f795c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:01.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:01.477+0000 7f7961a6e700 1 -- 172.21.15.88:0/372327333 shutdown_connections 2024-02-19T22:15:01.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:01.477+0000 7f7961a6e700 1 -- 172.21.15.88:0/372327333 wait complete. 2024-02-19T22:15:01.480 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:15:01.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:01 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:14:59.666024+0000 mgr.smithi052.ghqrtk (mgr.14162) 163 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:02.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:02 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:15:01.476478+0000 mon.smithi052 (mon.0) 211 : audit [DBG] from='client.? 172.21.15.88:0/372327333' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:03.173 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:03.173 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:03.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:03 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:15:01.666346+0000 mgr.smithi052.ghqrtk (mgr.14162) 164 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:05.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:05.084+0000 7f85628c9700 1 -- 172.21.15.88:0/501734643 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f855c060280 msgr2=0x7f855c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:05.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:05.084+0000 7f85628c9700 1 --2- 172.21.15.88:0/501734643 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f855c060280 0x7f855c0f90f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f854c009fb0 tx=0x7f854c009b30).stop 2024-02-19T22:15:05.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:05.084+0000 7f85628c9700 1 -- 172.21.15.88:0/501734643 shutdown_connections 2024-02-19T22:15:05.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:05.084+0000 7f85628c9700 1 --2- 172.21.15.88:0/501734643 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f855c060280 0x7f855c0f90f0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:05.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:05.084+0000 7f85628c9700 1 -- 172.21.15.88:0/501734643 >> 172.21.15.88:0/501734643 conn(0x7f855c0f0b70 msgr2=0x7f855c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:05.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:05.084+0000 7f85628c9700 1 -- 172.21.15.88:0/501734643 shutdown_connections 2024-02-19T22:15:05.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:05.084+0000 7f85628c9700 1 -- 172.21.15.88:0/501734643 wait complete. 2024-02-19T22:15:05.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:05.085+0000 7f85628c9700 1 Processor -- start 2024-02-19T22:15:05.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:05.085+0000 7f85628c9700 1 -- start start 2024-02-19T22:15:05.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:05.085+0000 7f85628c9700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f855c060280 0x7f855c106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:05.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:05.085+0000 7f85628c9700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f855c106c70 con 0x7f855c060280 2024-02-19T22:15:05.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:05.085+0000 7f855bfff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f855c060280 0x7f855c106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:05.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:05.085+0000 7f855bfff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f855c060280 0x7f855c106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:55716/0 (socket says 172.21.15.88:55716) 2024-02-19T22:15:05.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:05.085+0000 7f855bfff700 1 -- 172.21.15.88:0/148636648 learned_addr learned my addr 172.21.15.88:0/148636648 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:15:05.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:05.085+0000 7f855bfff700 1 -- 172.21.15.88:0/148636648 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f854c005040 con 0x7f855c060280 2024-02-19T22:15:05.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:05.086+0000 7f855bfff700 1 --2- 172.21.15.88:0/148636648 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f855c060280 0x7f855c106780 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f854c0089f0 tx=0x7f854c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:05.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:05.086+0000 7f8558ff9700 1 -- 172.21.15.88:0/148636648 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f854c009320 con 0x7f855c060280 2024-02-19T22:15:05.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:05.086+0000 7f85628c9700 1 -- 172.21.15.88:0/148636648 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f855c106e70 con 0x7f855c060280 2024-02-19T22:15:05.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:05.086+0000 7f8558ff9700 1 -- 172.21.15.88:0/148636648 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f854c013380 con 0x7f855c060280 2024-02-19T22:15:05.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:05.086+0000 7f8558ff9700 1 -- 172.21.15.88:0/148636648 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f854c014e20 con 0x7f855c060280 2024-02-19T22:15:05.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:05.086+0000 7f85628c9700 1 -- 172.21.15.88:0/148636648 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f855c107300 con 0x7f855c060280 2024-02-19T22:15:05.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:05.087+0000 7f85628c9700 1 -- 172.21.15.88:0/148636648 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f855c045ad0 con 0x7f855c060280 2024-02-19T22:15:05.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:05.087+0000 7f8558ff9700 1 -- 172.21.15.88:0/148636648 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f854c0095f0 con 0x7f855c060280 2024-02-19T22:15:05.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:05.087+0000 7f8558ff9700 1 --2- 172.21.15.88:0/148636648 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f854402ff10 0x7f85440323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:05.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:05.087+0000 7f8558ff9700 1 -- 172.21.15.88:0/148636648 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f854c040ec0 con 0x7f855c060280 2024-02-19T22:15:05.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:05.087+0000 7f855b7fe700 1 --2- 172.21.15.88:0/148636648 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f854402ff10 0x7f85440323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:05.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:05.087+0000 7f855b7fe700 1 --2- 172.21.15.88:0/148636648 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f854402ff10 0x7f85440323d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f85500096a0 tx=0x7f8550006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:05.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:05.092+0000 7f8558ff9700 1 -- 172.21.15.88:0/148636648 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f854c014420 con 0x7f855c060280 2024-02-19T22:15:05.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:05.294+0000 7f85628c9700 1 -- 172.21.15.88:0/148636648 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f855c108150 con 0x7f855c060280 2024-02-19T22:15:05.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:05.294+0000 7f8558ff9700 1 -- 172.21.15.88:0/148636648 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f854c010230 con 0x7f855c060280 2024-02-19T22:15:05.297 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:15:05.297 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:05.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:05.296+0000 7f85628c9700 1 -- 172.21.15.88:0/148636648 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f854402ff10 msgr2=0x7f85440323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:05.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:05.296+0000 7f85628c9700 1 --2- 172.21.15.88:0/148636648 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f854402ff10 0x7f85440323d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f85500096a0 tx=0x7f8550006b40).stop 2024-02-19T22:15:05.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:05.296+0000 7f85628c9700 1 -- 172.21.15.88:0/148636648 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f855c060280 msgr2=0x7f855c106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:05.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:05.296+0000 7f85628c9700 1 --2- 172.21.15.88:0/148636648 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f855c060280 0x7f855c106780 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f854c0089f0 tx=0x7f854c00b040).stop 2024-02-19T22:15:05.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:05.296+0000 7f85628c9700 1 -- 172.21.15.88:0/148636648 shutdown_connections 2024-02-19T22:15:05.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:05.296+0000 7f85628c9700 1 --2- 172.21.15.88:0/148636648 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f854402ff10 0x7f85440323d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:05.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:05.296+0000 7f85628c9700 1 --2- 172.21.15.88:0/148636648 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f855c060280 0x7f855c106780 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:05.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:05.296+0000 7f85628c9700 1 -- 172.21.15.88:0/148636648 >> 172.21.15.88:0/148636648 conn(0x7f855c0f0b70 msgr2=0x7f855c0f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:05.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:05.296+0000 7f85628c9700 1 -- 172.21.15.88:0/148636648 shutdown_connections 2024-02-19T22:15:05.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:05.296+0000 7f85628c9700 1 -- 172.21.15.88:0/148636648 wait complete. 2024-02-19T22:15:05.299 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:15:05.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:05 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:15:03.666700+0000 mgr.smithi052.ghqrtk (mgr.14162) 165 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:06.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:06 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:15:05.295965+0000 mon.smithi052 (mon.0) 212 : audit [DBG] from='client.? 172.21.15.88:0/148636648' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:06.854 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:06.854 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:07.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:07 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:15:05.666966+0000 mgr.smithi052.ghqrtk (mgr.14162) 166 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:08.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:08.635+0000 7f5100689700 1 -- 172.21.15.88:0/1586821889 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50f80f6510 msgr2=0x7f50f80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:08.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:08.635+0000 7f5100689700 1 --2- 172.21.15.88:0/1586821889 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50f80f6510 0x7f50f80f6930 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f50f4009fb0 tx=0x7f50f4009b30).stop 2024-02-19T22:15:08.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:08.635+0000 7f5100689700 1 -- 172.21.15.88:0/1586821889 shutdown_connections 2024-02-19T22:15:08.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:08.635+0000 7f5100689700 1 --2- 172.21.15.88:0/1586821889 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50f80f6510 0x7f50f80f6930 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:08.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:08.635+0000 7f5100689700 1 -- 172.21.15.88:0/1586821889 >> 172.21.15.88:0/1586821889 conn(0x7f50f80f1c70 msgr2=0x7f50f80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:08.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:08.636+0000 7f5100689700 1 -- 172.21.15.88:0/1586821889 shutdown_connections 2024-02-19T22:15:08.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:08.636+0000 7f5100689700 1 -- 172.21.15.88:0/1586821889 wait complete. 2024-02-19T22:15:08.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:08.636+0000 7f5100689700 1 Processor -- start 2024-02-19T22:15:08.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:08.636+0000 7f5100689700 1 -- start start 2024-02-19T22:15:08.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:08.636+0000 7f5100689700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50f80f6510 0x7f50f806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:08.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:08.636+0000 7f5100689700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50f8069e20 con 0x7f50f80f6510 2024-02-19T22:15:08.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:08.636+0000 7f50fe425700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50f80f6510 0x7f50f806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:08.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:08.636+0000 7f50fe425700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50f80f6510 0x7f50f806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:55740/0 (socket says 172.21.15.88:55740) 2024-02-19T22:15:08.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:08.636+0000 7f50fe425700 1 -- 172.21.15.88:0/269512939 learned_addr learned my addr 172.21.15.88:0/269512939 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:15:08.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:08.637+0000 7f50fe425700 1 -- 172.21.15.88:0/269512939 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50f4005040 con 0x7f50f80f6510 2024-02-19T22:15:08.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:08.637+0000 7f50fe425700 1 --2- 172.21.15.88:0/269512939 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50f80f6510 0x7f50f806b9a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f50f4005010 tx=0x7f50f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:08.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:08.637+0000 7f50eeffd700 1 -- 172.21.15.88:0/269512939 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50f4009320 con 0x7f50f80f6510 2024-02-19T22:15:08.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:08.637+0000 7f50eeffd700 1 -- 172.21.15.88:0/269512939 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f50f4013380 con 0x7f50f80f6510 2024-02-19T22:15:08.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:08.637+0000 7f5100689700 1 -- 172.21.15.88:0/269512939 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50f8069fc0 con 0x7f50f80f6510 2024-02-19T22:15:08.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:08.637+0000 7f50eeffd700 1 -- 172.21.15.88:0/269512939 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50f4014e50 con 0x7f50f80f6510 2024-02-19T22:15:08.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:08.637+0000 7f5100689700 1 -- 172.21.15.88:0/269512939 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50f806a450 con 0x7f50f80f6510 2024-02-19T22:15:08.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:08.637+0000 7f50eeffd700 1 -- 172.21.15.88:0/269512939 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f50f4014420 con 0x7f50f80f6510 2024-02-19T22:15:08.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:08.638+0000 7f50eeffd700 1 --2- 172.21.15.88:0/269512939 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f50e402ff10 0x7f50e40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:08.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:08.638+0000 7f50eeffd700 1 -- 172.21.15.88:0/269512939 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f50f4040f40 con 0x7f50f80f6510 2024-02-19T22:15:08.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:08.638+0000 7f5100689700 1 -- 172.21.15.88:0/269512939 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50f8059100 con 0x7f50f80f6510 2024-02-19T22:15:08.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:08.638+0000 7f50fdc24700 1 --2- 172.21.15.88:0/269512939 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f50e402ff10 0x7f50e40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:08.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:08.638+0000 7f50fdc24700 1 --2- 172.21.15.88:0/269512939 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f50e402ff10 0x7f50e40323d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f50e80096a0 tx=0x7f50e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:08.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:08.642+0000 7f50eeffd700 1 -- 172.21.15.88:0/269512939 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f50f40095f0 con 0x7f50f80f6510 2024-02-19T22:15:08.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:08.843+0000 7f5100689700 1 -- 172.21.15.88:0/269512939 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f50f806ab50 con 0x7f50f80f6510 2024-02-19T22:15:08.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:08.844+0000 7f50eeffd700 1 -- 172.21.15.88:0/269512939 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f50f402b020 con 0x7f50f80f6510 2024-02-19T22:15:08.846 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:15:08.846 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:08.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:08.846+0000 7f5100689700 1 -- 172.21.15.88:0/269512939 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f50e402ff10 msgr2=0x7f50e40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:08.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:08.846+0000 7f5100689700 1 --2- 172.21.15.88:0/269512939 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f50e402ff10 0x7f50e40323d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f50e80096a0 tx=0x7f50e8006b40).stop 2024-02-19T22:15:08.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:08.846+0000 7f5100689700 1 -- 172.21.15.88:0/269512939 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50f80f6510 msgr2=0x7f50f806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:08.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:08.846+0000 7f5100689700 1 --2- 172.21.15.88:0/269512939 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50f80f6510 0x7f50f806b9a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f50f4005010 tx=0x7f50f400b040).stop 2024-02-19T22:15:08.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:08.846+0000 7f5100689700 1 -- 172.21.15.88:0/269512939 shutdown_connections 2024-02-19T22:15:08.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:08.846+0000 7f5100689700 1 --2- 172.21.15.88:0/269512939 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f50e402ff10 0x7f50e40323d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:08.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:08.846+0000 7f5100689700 1 --2- 172.21.15.88:0/269512939 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50f80f6510 0x7f50f806b9a0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:08.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:08.846+0000 7f5100689700 1 -- 172.21.15.88:0/269512939 >> 172.21.15.88:0/269512939 conn(0x7f50f80f1c70 msgr2=0x7f50f80f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:08.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:08.846+0000 7f5100689700 1 -- 172.21.15.88:0/269512939 shutdown_connections 2024-02-19T22:15:08.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:08.846+0000 7f5100689700 1 -- 172.21.15.88:0/269512939 wait complete. 2024-02-19T22:15:08.849 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:15:09.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:09 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:15:07.667221+0000 mgr.smithi052.ghqrtk (mgr.14162) 167 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:09.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:09 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:15:08.845590+0000 mon.smithi052 (mon.0) 213 : audit [DBG] from='client.? 172.21.15.88:0/269512939' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:10.432 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:10.432 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:11.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:11 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:15:09.667454+0000 mgr.smithi052.ghqrtk (mgr.14162) 168 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:12.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:12.269+0000 7f95cdcd9700 1 -- 172.21.15.88:0/3438341237 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f95c8060280 msgr2=0x7f95c80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:12.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:12.269+0000 7f95cdcd9700 1 --2- 172.21.15.88:0/3438341237 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f95c8060280 0x7f95c80f90f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f95bc009fb0 tx=0x7f95bc009b30).stop 2024-02-19T22:15:12.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:12.269+0000 7f95cdcd9700 1 -- 172.21.15.88:0/3438341237 shutdown_connections 2024-02-19T22:15:12.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:12.269+0000 7f95cdcd9700 1 --2- 172.21.15.88:0/3438341237 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f95c8060280 0x7f95c80f90f0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:12.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:12.269+0000 7f95cdcd9700 1 -- 172.21.15.88:0/3438341237 >> 172.21.15.88:0/3438341237 conn(0x7f95c80f0b70 msgr2=0x7f95c80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:12.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:12.269+0000 7f95cdcd9700 1 -- 172.21.15.88:0/3438341237 shutdown_connections 2024-02-19T22:15:12.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:12.269+0000 7f95cdcd9700 1 -- 172.21.15.88:0/3438341237 wait complete. 2024-02-19T22:15:12.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:12.269+0000 7f95cdcd9700 1 Processor -- start 2024-02-19T22:15:12.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:12.269+0000 7f95cdcd9700 1 -- start start 2024-02-19T22:15:12.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:12.270+0000 7f95cdcd9700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f95c8060280 0x7f95c8106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:12.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:12.270+0000 7f95cdcd9700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c8106c70 con 0x7f95c8060280 2024-02-19T22:15:12.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:12.270+0000 7f95c77fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f95c8060280 0x7f95c8106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:12.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:12.270+0000 7f95c77fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f95c8060280 0x7f95c8106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:55760/0 (socket says 172.21.15.88:55760) 2024-02-19T22:15:12.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:12.270+0000 7f95c77fe700 1 -- 172.21.15.88:0/4054869094 learned_addr learned my addr 172.21.15.88:0/4054869094 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:15:12.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:12.270+0000 7f95c77fe700 1 -- 172.21.15.88:0/4054869094 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95bc005040 con 0x7f95c8060280 2024-02-19T22:15:12.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:12.270+0000 7f95c77fe700 1 --2- 172.21.15.88:0/4054869094 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f95c8060280 0x7f95c8106780 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f95bc0089f0 tx=0x7f95bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:12.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:12.270+0000 7f95affff700 1 -- 172.21.15.88:0/4054869094 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95bc009320 con 0x7f95c8060280 2024-02-19T22:15:12.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:12.270+0000 7f95cdcd9700 1 -- 172.21.15.88:0/4054869094 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95c8106e70 con 0x7f95c8060280 2024-02-19T22:15:12.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:12.270+0000 7f95affff700 1 -- 172.21.15.88:0/4054869094 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f95bc013380 con 0x7f95c8060280 2024-02-19T22:15:12.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:12.270+0000 7f95affff700 1 -- 172.21.15.88:0/4054869094 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95bc014e20 con 0x7f95c8060280 2024-02-19T22:15:12.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:12.270+0000 7f95cdcd9700 1 -- 172.21.15.88:0/4054869094 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95c8107300 con 0x7f95c8060280 2024-02-19T22:15:12.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:12.271+0000 7f95affff700 1 -- 172.21.15.88:0/4054869094 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f95bc0263a0 con 0x7f95c8060280 2024-02-19T22:15:12.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:12.271+0000 7f95affff700 1 --2- 172.21.15.88:0/4054869094 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f95b002ff10 0x7f95b00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:12.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:12.271+0000 7f95affff700 1 -- 172.21.15.88:0/4054869094 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f95bc041010 con 0x7f95c8060280 2024-02-19T22:15:12.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:12.271+0000 7f95cdcd9700 1 -- 172.21.15.88:0/4054869094 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95c8059100 con 0x7f95c8060280 2024-02-19T22:15:12.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:12.271+0000 7f95c6ffd700 1 --2- 172.21.15.88:0/4054869094 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f95b002ff10 0x7f95b00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:12.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:12.272+0000 7f95c6ffd700 1 --2- 172.21.15.88:0/4054869094 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f95b002ff10 0x7f95b00323d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f95b80096a0 tx=0x7f95b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:12.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:12.276+0000 7f95affff700 1 -- 172.21.15.88:0/4054869094 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f95bc012920 con 0x7f95c8060280 2024-02-19T22:15:12.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:12.480+0000 7f95cdcd9700 1 -- 172.21.15.88:0/4054869094 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f95c8045ad0 con 0x7f95c8060280 2024-02-19T22:15:12.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:12.480+0000 7f95affff700 1 -- 172.21.15.88:0/4054869094 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f95bc02b020 con 0x7f95c8060280 2024-02-19T22:15:12.482 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:15:12.482 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:12.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:12.482+0000 7f95cdcd9700 1 -- 172.21.15.88:0/4054869094 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f95b002ff10 msgr2=0x7f95b00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:12.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:12.482+0000 7f95cdcd9700 1 --2- 172.21.15.88:0/4054869094 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f95b002ff10 0x7f95b00323d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f95b80096a0 tx=0x7f95b8006b40).stop 2024-02-19T22:15:12.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:12.482+0000 7f95cdcd9700 1 -- 172.21.15.88:0/4054869094 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f95c8060280 msgr2=0x7f95c8106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:12.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:12.482+0000 7f95cdcd9700 1 --2- 172.21.15.88:0/4054869094 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f95c8060280 0x7f95c8106780 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f95bc0089f0 tx=0x7f95bc00b040).stop 2024-02-19T22:15:12.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:12.482+0000 7f95cdcd9700 1 -- 172.21.15.88:0/4054869094 shutdown_connections 2024-02-19T22:15:12.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:12.482+0000 7f95cdcd9700 1 --2- 172.21.15.88:0/4054869094 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f95b002ff10 0x7f95b00323d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:12.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:12.482+0000 7f95cdcd9700 1 --2- 172.21.15.88:0/4054869094 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f95c8060280 0x7f95c8106780 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:12.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:12.482+0000 7f95cdcd9700 1 -- 172.21.15.88:0/4054869094 >> 172.21.15.88:0/4054869094 conn(0x7f95c80f0b70 msgr2=0x7f95c80f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:12.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:12.482+0000 7f95cdcd9700 1 -- 172.21.15.88:0/4054869094 shutdown_connections 2024-02-19T22:15:12.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:12.482+0000 7f95cdcd9700 1 -- 172.21.15.88:0/4054869094 wait complete. 2024-02-19T22:15:12.485 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:15:13.390 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:13 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:15:11.667672+0000 mgr.smithi052.ghqrtk (mgr.14162) 169 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:13.390 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:13 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:15:12.481866+0000 mon.smithi052 (mon.0) 214 : audit [DBG] from='client.? 172.21.15.88:0/4054869094' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:14.056 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:14.056 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:15.293 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:15 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:15:13.668032+0000 mgr.smithi052.ghqrtk (mgr.14162) 170 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:15.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:15.732+0000 7fa953138700 1 -- 172.21.15.88:0/3807841776 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa94c0f51a0 msgr2=0x7fa94c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:15.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:15.732+0000 7fa953138700 1 --2- 172.21.15.88:0/3807841776 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa94c0f51a0 0x7fa94c0f55c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7fa93c009fb0 tx=0x7fa93c009b30).stop 2024-02-19T22:15:15.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:15.732+0000 7fa953138700 1 -- 172.21.15.88:0/3807841776 shutdown_connections 2024-02-19T22:15:15.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:15.732+0000 7fa953138700 1 --2- 172.21.15.88:0/3807841776 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa94c0f51a0 0x7fa94c0f55c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:15.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:15.732+0000 7fa953138700 1 -- 172.21.15.88:0/3807841776 >> 172.21.15.88:0/3807841776 conn(0x7fa94c0f09a0 msgr2=0x7fa94c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:15.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:15.732+0000 7fa953138700 1 -- 172.21.15.88:0/3807841776 shutdown_connections 2024-02-19T22:15:15.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:15.732+0000 7fa953138700 1 -- 172.21.15.88:0/3807841776 wait complete. 2024-02-19T22:15:15.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:15.733+0000 7fa953138700 1 Processor -- start 2024-02-19T22:15:15.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:15.733+0000 7fa953138700 1 -- start start 2024-02-19T22:15:15.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:15.733+0000 7fa953138700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa94c0f51a0 0x7fa94c1070c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:15.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:15.733+0000 7fa953138700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa94c060240 con 0x7fa94c0f51a0 2024-02-19T22:15:15.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:15.733+0000 7fa952136700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa94c0f51a0 0x7fa94c1070c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:15.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:15.733+0000 7fa952136700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa94c0f51a0 0x7fa94c1070c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:34232/0 (socket says 172.21.15.88:34232) 2024-02-19T22:15:15.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:15.733+0000 7fa952136700 1 -- 172.21.15.88:0/2050682981 learned_addr learned my addr 172.21.15.88:0/2050682981 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:15:15.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:15.733+0000 7fa952136700 1 -- 172.21.15.88:0/2050682981 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa93c005040 con 0x7fa94c0f51a0 2024-02-19T22:15:15.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:15.734+0000 7fa952136700 1 --2- 172.21.15.88:0/2050682981 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa94c0f51a0 0x7fa94c1070c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fa93c0089f0 tx=0x7fa93c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:15.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:15.734+0000 7fa942ffd700 1 -- 172.21.15.88:0/2050682981 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa93c009320 con 0x7fa94c0f51a0 2024-02-19T22:15:15.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:15.734+0000 7fa942ffd700 1 -- 172.21.15.88:0/2050682981 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa93c013380 con 0x7fa94c0f51a0 2024-02-19T22:15:15.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:15.734+0000 7fa953138700 1 -- 172.21.15.88:0/2050682981 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa94c060440 con 0x7fa94c0f51a0 2024-02-19T22:15:15.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:15.734+0000 7fa942ffd700 1 -- 172.21.15.88:0/2050682981 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa93c014e20 con 0x7fa94c0f51a0 2024-02-19T22:15:15.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:15.734+0000 7fa953138700 1 -- 172.21.15.88:0/2050682981 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa94c1077f0 con 0x7fa94c0f51a0 2024-02-19T22:15:15.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:15.734+0000 7fa942ffd700 1 -- 172.21.15.88:0/2050682981 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa93c014420 con 0x7fa94c0f51a0 2024-02-19T22:15:15.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:15.735+0000 7fa953138700 1 -- 172.21.15.88:0/2050682981 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa94c045ad0 con 0x7fa94c0f51a0 2024-02-19T22:15:15.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:15.735+0000 7fa942ffd700 1 --2- 172.21.15.88:0/2050682981 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa93802ff10 0x7fa9380323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:15.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:15.735+0000 7fa942ffd700 1 -- 172.21.15.88:0/2050682981 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa93c040f80 con 0x7fa94c0f51a0 2024-02-19T22:15:15.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:15.735+0000 7fa951935700 1 --2- 172.21.15.88:0/2050682981 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa93802ff10 0x7fa9380323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:15.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:15.735+0000 7fa951935700 1 --2- 172.21.15.88:0/2050682981 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa93802ff10 0x7fa9380323d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fa9480096a0 tx=0x7fa948006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:15.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:15.739+0000 7fa942ffd700 1 -- 172.21.15.88:0/2050682981 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa93c06d2b0 con 0x7fa94c0f51a0 2024-02-19T22:15:15.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:15.945+0000 7fa953138700 1 -- 172.21.15.88:0/2050682981 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa94c107f90 con 0x7fa94c0f51a0 2024-02-19T22:15:15.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:15.946+0000 7fa942ffd700 1 -- 172.21.15.88:0/2050682981 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa93c0095f0 con 0x7fa94c0f51a0 2024-02-19T22:15:15.948 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:15:15.948 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:15.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:15.948+0000 7fa953138700 1 -- 172.21.15.88:0/2050682981 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa93802ff10 msgr2=0x7fa9380323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:15.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:15.948+0000 7fa953138700 1 --2- 172.21.15.88:0/2050682981 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa93802ff10 0x7fa9380323d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fa9480096a0 tx=0x7fa948006b40).stop 2024-02-19T22:15:15.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:15.948+0000 7fa953138700 1 -- 172.21.15.88:0/2050682981 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa94c0f51a0 msgr2=0x7fa94c1070c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:15.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:15.948+0000 7fa953138700 1 --2- 172.21.15.88:0/2050682981 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa94c0f51a0 0x7fa94c1070c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fa93c0089f0 tx=0x7fa93c00b040).stop 2024-02-19T22:15:15.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:15.948+0000 7fa953138700 1 -- 172.21.15.88:0/2050682981 shutdown_connections 2024-02-19T22:15:15.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:15.948+0000 7fa953138700 1 --2- 172.21.15.88:0/2050682981 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa93802ff10 0x7fa9380323d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:15.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:15.948+0000 7fa953138700 1 --2- 172.21.15.88:0/2050682981 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa94c0f51a0 0x7fa94c1070c0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:15.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:15.948+0000 7fa953138700 1 -- 172.21.15.88:0/2050682981 >> 172.21.15.88:0/2050682981 conn(0x7fa94c0f09a0 msgr2=0x7fa94c0f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:15.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:15.948+0000 7fa953138700 1 -- 172.21.15.88:0/2050682981 shutdown_connections 2024-02-19T22:15:15.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:15.948+0000 7fa953138700 1 -- 172.21.15.88:0/2050682981 wait complete. 2024-02-19T22:15:15.951 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:15:16.544 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:16 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:15:15.947767+0000 mon.smithi052 (mon.0) 215 : audit [DBG] from='client.? 172.21.15.88:0/2050682981' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:17.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:17 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:15:15.668417+0000 mgr.smithi052.ghqrtk (mgr.14162) 171 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:17.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:17 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:15:16.304866+0000 mon.smithi052 (mon.0) 216 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' 2024-02-19T22:15:17.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:17 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cephadm 2024-02-19T22:15:16.306205+0000 mgr.smithi052.ghqrtk (mgr.14162) 172 : cephadm [INF] Deploying daemon node-exporter.smithi052 on smithi052 2024-02-19T22:15:17.669 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:17.669 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:19.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:19.465+0000 7f70d4c88700 1 -- 172.21.15.88:0/4209278254 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f70d0060220 msgr2=0x7f70d00fb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:19.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:19.465+0000 7f70d4c88700 1 --2- 172.21.15.88:0/4209278254 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f70d0060220 0x7f70d00fb550 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f70b8009fb0 tx=0x7f70b8009b30).stop 2024-02-19T22:15:19.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:19.465+0000 7f70d4c88700 1 -- 172.21.15.88:0/4209278254 shutdown_connections 2024-02-19T22:15:19.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:19.465+0000 7f70d4c88700 1 --2- 172.21.15.88:0/4209278254 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f70d0060220 0x7f70d00fb550 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:19.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:19.465+0000 7f70d4c88700 1 -- 172.21.15.88:0/4209278254 >> 172.21.15.88:0/4209278254 conn(0x7f70d00f0b70 msgr2=0x7f70d00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:19.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:19.465+0000 7f70d4c88700 1 -- 172.21.15.88:0/4209278254 shutdown_connections 2024-02-19T22:15:19.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:19.465+0000 7f70d4c88700 1 -- 172.21.15.88:0/4209278254 wait complete. 2024-02-19T22:15:19.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:19.465+0000 7f70d4c88700 1 Processor -- start 2024-02-19T22:15:19.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:19.466+0000 7f70d4c88700 1 -- start start 2024-02-19T22:15:19.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:19.466+0000 7f70d4c88700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f70d0060220 0x7f70d0108980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:19.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:19.466+0000 7f70d4c88700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70d0108e70 con 0x7f70d0060220 2024-02-19T22:15:19.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:19.466+0000 7f70ce59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f70d0060220 0x7f70d0108980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:19.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:19.466+0000 7f70ce59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f70d0060220 0x7f70d0108980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:34242/0 (socket says 172.21.15.88:34242) 2024-02-19T22:15:19.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:19.466+0000 7f70ce59c700 1 -- 172.21.15.88:0/4269491872 learned_addr learned my addr 172.21.15.88:0/4269491872 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:15:19.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:19.466+0000 7f70ce59c700 1 -- 172.21.15.88:0/4269491872 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70b8005040 con 0x7f70d0060220 2024-02-19T22:15:19.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:19.466+0000 7f70ce59c700 1 --2- 172.21.15.88:0/4269491872 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f70d0060220 0x7f70d0108980 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f70b8009830 tx=0x7f70b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:19.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:19.467+0000 7f70c6ffd700 1 -- 172.21.15.88:0/4269491872 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70b8009320 con 0x7f70d0060220 2024-02-19T22:15:19.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:19.467+0000 7f70d4c88700 1 -- 172.21.15.88:0/4269491872 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70d0109070 con 0x7f70d0060220 2024-02-19T22:15:19.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:19.467+0000 7f70c6ffd700 1 -- 172.21.15.88:0/4269491872 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f70b8013380 con 0x7f70d0060220 2024-02-19T22:15:19.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:19.467+0000 7f70c6ffd700 1 -- 172.21.15.88:0/4269491872 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70b8014420 con 0x7f70d0060220 2024-02-19T22:15:19.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:19.467+0000 7f70d4c88700 1 -- 172.21.15.88:0/4269491872 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70d0109500 con 0x7f70d0060220 2024-02-19T22:15:19.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:19.467+0000 7f70c6ffd700 1 -- 172.21.15.88:0/4269491872 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f70b8014580 con 0x7f70d0060220 2024-02-19T22:15:19.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:19.468+0000 7f70c6ffd700 1 --2- 172.21.15.88:0/4269491872 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f70bc02ff60 0x7f70bc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:19.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:19.468+0000 7f70d4c88700 1 -- 172.21.15.88:0/4269491872 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70d0045ad0 con 0x7f70d0060220 2024-02-19T22:15:19.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:19.468+0000 7f70c6ffd700 1 -- 172.21.15.88:0/4269491872 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f70b8040ed0 con 0x7f70d0060220 2024-02-19T22:15:19.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:19.468+0000 7f70cdd9b700 1 --2- 172.21.15.88:0/4269491872 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f70bc02ff60 0x7f70bc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:19.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:19.468+0000 7f70cdd9b700 1 --2- 172.21.15.88:0/4269491872 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f70bc02ff60 0x7f70bc032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f70c00096a0 tx=0x7f70c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:19.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:19.473+0000 7f70c6ffd700 1 -- 172.21.15.88:0/4269491872 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f70b80273f0 con 0x7f70d0060220 2024-02-19T22:15:19.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:19 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:15:17.668737+0000 mgr.smithi052.ghqrtk (mgr.14162) 173 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:19.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:19.678+0000 7f70d4c88700 1 -- 172.21.15.88:0/4269491872 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f70d00fe290 con 0x7f70d0060220 2024-02-19T22:15:19.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:19.678+0000 7f70c6ffd700 1 -- 172.21.15.88:0/4269491872 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f70b801c320 con 0x7f70d0060220 2024-02-19T22:15:19.681 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:15:19.681 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:19.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:19.680+0000 7f70d4c88700 1 -- 172.21.15.88:0/4269491872 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f70bc02ff60 msgr2=0x7f70bc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:19.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:19.680+0000 7f70d4c88700 1 --2- 172.21.15.88:0/4269491872 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f70bc02ff60 0x7f70bc032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f70c00096a0 tx=0x7f70c0006b40).stop 2024-02-19T22:15:19.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:19.680+0000 7f70d4c88700 1 -- 172.21.15.88:0/4269491872 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f70d0060220 msgr2=0x7f70d0108980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:19.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:19.680+0000 7f70d4c88700 1 --2- 172.21.15.88:0/4269491872 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f70d0060220 0x7f70d0108980 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f70b8009830 tx=0x7f70b800b040).stop 2024-02-19T22:15:19.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:19.680+0000 7f70d4c88700 1 -- 172.21.15.88:0/4269491872 shutdown_connections 2024-02-19T22:15:19.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:19.680+0000 7f70d4c88700 1 --2- 172.21.15.88:0/4269491872 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f70bc02ff60 0x7f70bc032420 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:19.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:19.680+0000 7f70d4c88700 1 --2- 172.21.15.88:0/4269491872 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f70d0060220 0x7f70d0108980 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:19.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:19.680+0000 7f70d4c88700 1 -- 172.21.15.88:0/4269491872 >> 172.21.15.88:0/4269491872 conn(0x7f70d00f0b70 msgr2=0x7f70d00f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:19.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:19.680+0000 7f70d4c88700 1 -- 172.21.15.88:0/4269491872 shutdown_connections 2024-02-19T22:15:19.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:19.680+0000 7f70d4c88700 1 -- 172.21.15.88:0/4269491872 wait complete. 2024-02-19T22:15:19.683 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:15:20.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:20 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:15:19.679921+0000 mon.smithi052 (mon.0) 217 : audit [DBG] from='client.? 172.21.15.88:0/4269491872' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:21.573 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:21.574 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:21.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:21 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:15:19.669016+0000 mgr.smithi052.ghqrtk (mgr.14162) 174 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:23.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:23 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:15:21.669359+0000 mgr.smithi052.ghqrtk (mgr.14162) 175 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:23.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:23.622+0000 7fce27ced700 1 -- 172.21.15.88:0/1564776225 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce200f5410 msgr2=0x7fce200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:23.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:23.622+0000 7fce27ced700 1 --2- 172.21.15.88:0/1564776225 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce200f5410 0x7fce200f5830 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7fce10009fb0 tx=0x7fce10009b30).stop 2024-02-19T22:15:23.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:23.622+0000 7fce27ced700 1 -- 172.21.15.88:0/1564776225 shutdown_connections 2024-02-19T22:15:23.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:23.622+0000 7fce27ced700 1 --2- 172.21.15.88:0/1564776225 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce200f5410 0x7fce200f5830 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:23.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:23.622+0000 7fce27ced700 1 -- 172.21.15.88:0/1564776225 >> 172.21.15.88:0/1564776225 conn(0x7fce200f0b70 msgr2=0x7fce200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:23.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:23.622+0000 7fce27ced700 1 -- 172.21.15.88:0/1564776225 shutdown_connections 2024-02-19T22:15:23.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:23.622+0000 7fce27ced700 1 -- 172.21.15.88:0/1564776225 wait complete. 2024-02-19T22:15:23.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:23.623+0000 7fce27ced700 1 Processor -- start 2024-02-19T22:15:23.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:23.623+0000 7fce27ced700 1 -- start start 2024-02-19T22:15:23.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:23.623+0000 7fce27ced700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce200f5410 0x7fce201056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:23.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:23.623+0000 7fce27ced700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce20105bd0 con 0x7fce200f5410 2024-02-19T22:15:23.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:23.623+0000 7fce25a89700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce200f5410 0x7fce201056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:23.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:23.623+0000 7fce25a89700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce200f5410 0x7fce201056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:59062/0 (socket says 172.21.15.88:59062) 2024-02-19T22:15:23.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:23.623+0000 7fce25a89700 1 -- 172.21.15.88:0/162732657 learned_addr learned my addr 172.21.15.88:0/162732657 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:15:23.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:23.623+0000 7fce25a89700 1 -- 172.21.15.88:0/162732657 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce10005040 con 0x7fce200f5410 2024-02-19T22:15:23.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:23.623+0000 7fce25a89700 1 --2- 172.21.15.88:0/162732657 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce200f5410 0x7fce201056e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7fce10005010 tx=0x7fce1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:23.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:23.624+0000 7fce167fc700 1 -- 172.21.15.88:0/162732657 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce10009320 con 0x7fce200f5410 2024-02-19T22:15:23.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:23.624+0000 7fce167fc700 1 -- 172.21.15.88:0/162732657 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fce10013380 con 0x7fce200f5410 2024-02-19T22:15:23.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:23.624+0000 7fce27ced700 1 -- 172.21.15.88:0/162732657 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce20105dd0 con 0x7fce200f5410 2024-02-19T22:15:23.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:23.624+0000 7fce167fc700 1 -- 172.21.15.88:0/162732657 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce10014e50 con 0x7fce200f5410 2024-02-19T22:15:23.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:23.624+0000 7fce27ced700 1 -- 172.21.15.88:0/162732657 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce20106260 con 0x7fce200f5410 2024-02-19T22:15:23.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:23.624+0000 7fce167fc700 1 -- 172.21.15.88:0/162732657 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fce100095f0 con 0x7fce200f5410 2024-02-19T22:15:23.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:23.624+0000 7fce27ced700 1 -- 172.21.15.88:0/162732657 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce20045ad0 con 0x7fce200f5410 2024-02-19T22:15:23.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:23.625+0000 7fce167fc700 1 --2- 172.21.15.88:0/162732657 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fce0c02ff10 0x7fce0c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:23.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:23.625+0000 7fce167fc700 1 -- 172.21.15.88:0/162732657 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fce10040e90 con 0x7fce200f5410 2024-02-19T22:15:23.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:23.625+0000 7fce25288700 1 --2- 172.21.15.88:0/162732657 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fce0c02ff10 0x7fce0c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:23.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:23.625+0000 7fce25288700 1 --2- 172.21.15.88:0/162732657 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fce0c02ff10 0x7fce0c0323d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fce1c0096a0 tx=0x7fce1c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:23.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:23.630+0000 7fce167fc700 1 -- 172.21.15.88:0/162732657 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fce10014420 con 0x7fce200f5410 2024-02-19T22:15:23.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:23.834+0000 7fce27ced700 1 -- 172.21.15.88:0/162732657 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fce20107110 con 0x7fce200f5410 2024-02-19T22:15:23.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:23.835+0000 7fce167fc700 1 -- 172.21.15.88:0/162732657 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fce10010230 con 0x7fce200f5410 2024-02-19T22:15:23.837 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:15:23.837 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:23.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:23.836+0000 7fce27ced700 1 -- 172.21.15.88:0/162732657 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fce0c02ff10 msgr2=0x7fce0c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:23.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:23.836+0000 7fce27ced700 1 --2- 172.21.15.88:0/162732657 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fce0c02ff10 0x7fce0c0323d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fce1c0096a0 tx=0x7fce1c006b40).stop 2024-02-19T22:15:23.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:23.836+0000 7fce27ced700 1 -- 172.21.15.88:0/162732657 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce200f5410 msgr2=0x7fce201056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:23.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:23.836+0000 7fce27ced700 1 --2- 172.21.15.88:0/162732657 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce200f5410 0x7fce201056e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7fce10005010 tx=0x7fce1000b040).stop 2024-02-19T22:15:23.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:23.836+0000 7fce27ced700 1 -- 172.21.15.88:0/162732657 shutdown_connections 2024-02-19T22:15:23.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:23.836+0000 7fce27ced700 1 --2- 172.21.15.88:0/162732657 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fce0c02ff10 0x7fce0c0323d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:23.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:23.836+0000 7fce27ced700 1 --2- 172.21.15.88:0/162732657 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce200f5410 0x7fce201056e0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:23.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:23.836+0000 7fce27ced700 1 -- 172.21.15.88:0/162732657 >> 172.21.15.88:0/162732657 conn(0x7fce200f0b70 msgr2=0x7fce200f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:23.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:23.837+0000 7fce27ced700 1 -- 172.21.15.88:0/162732657 shutdown_connections 2024-02-19T22:15:23.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:23.837+0000 7fce27ced700 1 -- 172.21.15.88:0/162732657 wait complete. 2024-02-19T22:15:23.840 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:15:24.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:24 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:15:23.836037+0000 mon.smithi052 (mon.0) 218 : audit [DBG] from='client.? 172.21.15.88:0/162732657' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:25.372 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:25.372 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:25.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:25 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:15:23.669611+0000 mgr.smithi052.ghqrtk (mgr.14162) 176 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:27.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:27.264+0000 7f4f313e3700 1 -- 172.21.15.88:0/902652105 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f2c0f5410 msgr2=0x7f4f2c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:27.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:27.264+0000 7f4f313e3700 1 --2- 172.21.15.88:0/902652105 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f2c0f5410 0x7f4f2c0f5830 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f4f14009fb0 tx=0x7f4f14009b30).stop 2024-02-19T22:15:27.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:27.264+0000 7f4f313e3700 1 -- 172.21.15.88:0/902652105 shutdown_connections 2024-02-19T22:15:27.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:27.264+0000 7f4f313e3700 1 --2- 172.21.15.88:0/902652105 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f2c0f5410 0x7f4f2c0f5830 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:27.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:27.264+0000 7f4f313e3700 1 -- 172.21.15.88:0/902652105 >> 172.21.15.88:0/902652105 conn(0x7f4f2c0f0b70 msgr2=0x7f4f2c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:27.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:27.264+0000 7f4f313e3700 1 -- 172.21.15.88:0/902652105 shutdown_connections 2024-02-19T22:15:27.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:27.264+0000 7f4f313e3700 1 -- 172.21.15.88:0/902652105 wait complete. 2024-02-19T22:15:27.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:27.265+0000 7f4f313e3700 1 Processor -- start 2024-02-19T22:15:27.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:27.265+0000 7f4f313e3700 1 -- start start 2024-02-19T22:15:27.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:27.265+0000 7f4f313e3700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f2c0f5410 0x7f4f2c10b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:27.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:27.265+0000 7f4f313e3700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2c1072f0 con 0x7f4f2c0f5410 2024-02-19T22:15:27.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:27.265+0000 7f4f2affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f2c0f5410 0x7f4f2c10b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:27.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:27.265+0000 7f4f2affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f2c0f5410 0x7f4f2c10b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:59082/0 (socket says 172.21.15.88:59082) 2024-02-19T22:15:27.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:27.265+0000 7f4f2affd700 1 -- 172.21.15.88:0/2779853439 learned_addr learned my addr 172.21.15.88:0/2779853439 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:15:27.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:27.266+0000 7f4f2affd700 1 -- 172.21.15.88:0/2779853439 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f14005040 con 0x7f4f2c0f5410 2024-02-19T22:15:27.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:27.266+0000 7f4f2affd700 1 --2- 172.21.15.88:0/2779853439 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f2c0f5410 0x7f4f2c10b900 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f4f14005010 tx=0x7f4f1400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:27.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:27.266+0000 7f4f237fe700 1 -- 172.21.15.88:0/2779853439 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f14009320 con 0x7f4f2c0f5410 2024-02-19T22:15:27.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:27.266+0000 7f4f237fe700 1 -- 172.21.15.88:0/2779853439 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4f14013380 con 0x7f4f2c0f5410 2024-02-19T22:15:27.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:27.266+0000 7f4f313e3700 1 -- 172.21.15.88:0/2779853439 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f2c060220 con 0x7f4f2c0f5410 2024-02-19T22:15:27.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:27.266+0000 7f4f237fe700 1 -- 172.21.15.88:0/2779853439 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f14014e50 con 0x7f4f2c0f5410 2024-02-19T22:15:27.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:27.266+0000 7f4f313e3700 1 -- 172.21.15.88:0/2779853439 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f2c10bdf0 con 0x7f4f2c0f5410 2024-02-19T22:15:27.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:27.267+0000 7f4f237fe700 1 -- 172.21.15.88:0/2779853439 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4f140095f0 con 0x7f4f2c0f5410 2024-02-19T22:15:27.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:27.267+0000 7f4f237fe700 1 --2- 172.21.15.88:0/2779853439 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f4f1802ff10 0x7f4f180323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:27.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:27.267+0000 7f4f313e3700 1 -- 172.21.15.88:0/2779853439 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f2c045ad0 con 0x7f4f2c0f5410 2024-02-19T22:15:27.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:27.267+0000 7f4f237fe700 1 -- 172.21.15.88:0/2779853439 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4f14040ee0 con 0x7f4f2c0f5410 2024-02-19T22:15:27.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:27.267+0000 7f4f2a7fc700 1 --2- 172.21.15.88:0/2779853439 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f4f1802ff10 0x7f4f180323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:27.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:27.268+0000 7f4f2a7fc700 1 --2- 172.21.15.88:0/2779853439 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f4f1802ff10 0x7f4f180323d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f4f1c0096a0 tx=0x7f4f1c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:27.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:27.272+0000 7f4f237fe700 1 -- 172.21.15.88:0/2779853439 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4f14023250 con 0x7f4f2c0f5410 2024-02-19T22:15:27.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:27.476+0000 7f4f313e3700 1 -- 172.21.15.88:0/2779853439 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4f2c10c5b0 con 0x7f4f2c0f5410 2024-02-19T22:15:27.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:27.477+0000 7f4f237fe700 1 -- 172.21.15.88:0/2779853439 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4f14023250 con 0x7f4f2c0f5410 2024-02-19T22:15:27.480 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:15:27.480 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:27.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:27.478+0000 7f4f313e3700 1 -- 172.21.15.88:0/2779853439 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f4f1802ff10 msgr2=0x7f4f180323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:27.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:27.478+0000 7f4f313e3700 1 --2- 172.21.15.88:0/2779853439 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f4f1802ff10 0x7f4f180323d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f4f1c0096a0 tx=0x7f4f1c006b40).stop 2024-02-19T22:15:27.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:27.478+0000 7f4f313e3700 1 -- 172.21.15.88:0/2779853439 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f2c0f5410 msgr2=0x7f4f2c10b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:27.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:27.478+0000 7f4f313e3700 1 --2- 172.21.15.88:0/2779853439 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f2c0f5410 0x7f4f2c10b900 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f4f14005010 tx=0x7f4f1400b040).stop 2024-02-19T22:15:27.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:27.478+0000 7f4f313e3700 1 -- 172.21.15.88:0/2779853439 shutdown_connections 2024-02-19T22:15:27.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:27.478+0000 7f4f313e3700 1 --2- 172.21.15.88:0/2779853439 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f4f1802ff10 0x7f4f180323d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:27.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:27.478+0000 7f4f313e3700 1 --2- 172.21.15.88:0/2779853439 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f2c0f5410 0x7f4f2c10b900 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:27.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:27.478+0000 7f4f313e3700 1 -- 172.21.15.88:0/2779853439 >> 172.21.15.88:0/2779853439 conn(0x7f4f2c0f0b70 msgr2=0x7f4f2c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:27.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:27.478+0000 7f4f313e3700 1 -- 172.21.15.88:0/2779853439 shutdown_connections 2024-02-19T22:15:27.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:27.478+0000 7f4f313e3700 1 -- 172.21.15.88:0/2779853439 wait complete. 2024-02-19T22:15:27.481 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:15:27.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:27 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:15:25.669887+0000 mgr.smithi052.ghqrtk (mgr.14162) 177 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:28.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:28 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:15:27.477210+0000 mon.smithi052 (mon.0) 219 : audit [DBG] from='client.? 172.21.15.88:0/2779853439' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:29.129 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:29.129 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:29.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:29 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:15:27.670215+0000 mgr.smithi052.ghqrtk (mgr.14162) 178 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:30.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:30.892+0000 7f7bdbd07700 1 -- 172.21.15.88:0/3289897172 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7bd40f47c0 msgr2=0x7f7bd40f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:30.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:30.892+0000 7f7bdbd07700 1 --2- 172.21.15.88:0/3289897172 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7bd40f47c0 0x7f7bd40f4be0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f7bc8009fb0 tx=0x7f7bc8009b30).stop 2024-02-19T22:15:30.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:30.892+0000 7f7bdbd07700 1 -- 172.21.15.88:0/3289897172 shutdown_connections 2024-02-19T22:15:30.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:30.892+0000 7f7bdbd07700 1 --2- 172.21.15.88:0/3289897172 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7bd40f47c0 0x7f7bd40f4be0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:30.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:30.892+0000 7f7bdbd07700 1 -- 172.21.15.88:0/3289897172 >> 172.21.15.88:0/3289897172 conn(0x7f7bd40f02e0 msgr2=0x7f7bd40f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:30.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:30.892+0000 7f7bdbd07700 1 -- 172.21.15.88:0/3289897172 shutdown_connections 2024-02-19T22:15:30.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:30.892+0000 7f7bdbd07700 1 -- 172.21.15.88:0/3289897172 wait complete. 2024-02-19T22:15:30.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:30.893+0000 7f7bdbd07700 1 Processor -- start 2024-02-19T22:15:30.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:30.893+0000 7f7bdbd07700 1 -- start start 2024-02-19T22:15:30.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:30.893+0000 7f7bdbd07700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7bd40f47c0 0x7f7bd41067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:30.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:30.893+0000 7f7bdbd07700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd4106cc0 con 0x7f7bd40f47c0 2024-02-19T22:15:30.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:30.893+0000 7f7bd9aa3700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7bd40f47c0 0x7f7bd41067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:30.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:30.893+0000 7f7bd9aa3700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7bd40f47c0 0x7f7bd41067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:59108/0 (socket says 172.21.15.88:59108) 2024-02-19T22:15:30.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:30.893+0000 7f7bd9aa3700 1 -- 172.21.15.88:0/2895562573 learned_addr learned my addr 172.21.15.88:0/2895562573 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:15:30.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:30.893+0000 7f7bd9aa3700 1 -- 172.21.15.88:0/2895562573 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bc8005040 con 0x7f7bd40f47c0 2024-02-19T22:15:30.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:30.893+0000 7f7bd9aa3700 1 --2- 172.21.15.88:0/2895562573 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7bd40f47c0 0x7f7bd41067d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f7bc8005010 tx=0x7f7bc800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:30.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:30.894+0000 7f7bc67fc700 1 -- 172.21.15.88:0/2895562573 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bc8009320 con 0x7f7bd40f47c0 2024-02-19T22:15:30.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:30.894+0000 7f7bdbd07700 1 -- 172.21.15.88:0/2895562573 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bd4106ec0 con 0x7f7bd40f47c0 2024-02-19T22:15:30.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:30.894+0000 7f7bc67fc700 1 -- 172.21.15.88:0/2895562573 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7bc8013380 con 0x7f7bd40f47c0 2024-02-19T22:15:30.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:30.894+0000 7f7bc67fc700 1 -- 172.21.15.88:0/2895562573 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bc8014e50 con 0x7f7bd40f47c0 2024-02-19T22:15:30.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:30.894+0000 7f7bdbd07700 1 -- 172.21.15.88:0/2895562573 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bd4107350 con 0x7f7bd40f47c0 2024-02-19T22:15:30.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:30.894+0000 7f7bc67fc700 1 -- 172.21.15.88:0/2895562573 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7bc80263a0 con 0x7f7bd40f47c0 2024-02-19T22:15:30.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:30.895+0000 7f7bc67fc700 1 --2- 172.21.15.88:0/2895562573 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f7bc002ff10 0x7f7bc00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:30.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:30.895+0000 7f7bc67fc700 1 -- 172.21.15.88:0/2895562573 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7bc8040f00 con 0x7f7bd40f47c0 2024-02-19T22:15:30.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:30.895+0000 7f7bdbd07700 1 -- 172.21.15.88:0/2895562573 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bd4045ad0 con 0x7f7bd40f47c0 2024-02-19T22:15:30.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:30.895+0000 7f7bd92a2700 1 --2- 172.21.15.88:0/2895562573 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f7bc002ff10 0x7f7bc00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:30.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:30.895+0000 7f7bd92a2700 1 --2- 172.21.15.88:0/2895562573 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f7bc002ff10 0x7f7bc00323d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f7bd00096a0 tx=0x7f7bd0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:30.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:30.899+0000 7f7bc67fc700 1 -- 172.21.15.88:0/2895562573 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7bc8012920 con 0x7f7bd40f47c0 2024-02-19T22:15:31.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:31.106+0000 7f7bdbd07700 1 -- 172.21.15.88:0/2895562573 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7bd41080a0 con 0x7f7bd40f47c0 2024-02-19T22:15:31.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:31.106+0000 7f7bc67fc700 1 -- 172.21.15.88:0/2895562573 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7bc802b020 con 0x7f7bd40f47c0 2024-02-19T22:15:31.108 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:15:31.108 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:31.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:31.108+0000 7f7bdbd07700 1 -- 172.21.15.88:0/2895562573 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f7bc002ff10 msgr2=0x7f7bc00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:31.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:31.108+0000 7f7bdbd07700 1 --2- 172.21.15.88:0/2895562573 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f7bc002ff10 0x7f7bc00323d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f7bd00096a0 tx=0x7f7bd0006b40).stop 2024-02-19T22:15:31.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:31.108+0000 7f7bdbd07700 1 -- 172.21.15.88:0/2895562573 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7bd40f47c0 msgr2=0x7f7bd41067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:31.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:31.108+0000 7f7bdbd07700 1 --2- 172.21.15.88:0/2895562573 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7bd40f47c0 0x7f7bd41067d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f7bc8005010 tx=0x7f7bc800b040).stop 2024-02-19T22:15:31.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:31.108+0000 7f7bdbd07700 1 -- 172.21.15.88:0/2895562573 shutdown_connections 2024-02-19T22:15:31.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:31.108+0000 7f7bdbd07700 1 --2- 172.21.15.88:0/2895562573 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f7bc002ff10 0x7f7bc00323d0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:31.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:31.108+0000 7f7bdbd07700 1 --2- 172.21.15.88:0/2895562573 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7bd40f47c0 0x7f7bd41067d0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:31.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:31.108+0000 7f7bdbd07700 1 -- 172.21.15.88:0/2895562573 >> 172.21.15.88:0/2895562573 conn(0x7f7bd40f02e0 msgr2=0x7f7bd40f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:31.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:31.108+0000 7f7bdbd07700 1 -- 172.21.15.88:0/2895562573 shutdown_connections 2024-02-19T22:15:31.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:31.108+0000 7f7bdbd07700 1 -- 172.21.15.88:0/2895562573 wait complete. 2024-02-19T22:15:31.110 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:15:31.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:31 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:15:29.670617+0000 mgr.smithi052.ghqrtk (mgr.14162) 179 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:31.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:31 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:15:31.106932+0000 mon.smithi052 (mon.0) 220 : audit [DBG] from='client.? 172.21.15.88:0/2895562573' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:32.881 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:32.881 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:33.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:33 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:15:31.670930+0000 mgr.smithi052.ghqrtk (mgr.14162) 180 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:34.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:34.802+0000 7fdf41462700 1 -- 172.21.15.88:0/802168127 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf3c0f6510 msgr2=0x7fdf3c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:34.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:34.802+0000 7fdf41462700 1 --2- 172.21.15.88:0/802168127 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf3c0f6510 0x7fdf3c0f6930 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7fdf2c009fb0 tx=0x7fdf2c009b30).stop 2024-02-19T22:15:34.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:34.803+0000 7fdf41462700 1 -- 172.21.15.88:0/802168127 shutdown_connections 2024-02-19T22:15:34.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:34.803+0000 7fdf41462700 1 --2- 172.21.15.88:0/802168127 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf3c0f6510 0x7fdf3c0f6930 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:34.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:34.803+0000 7fdf41462700 1 -- 172.21.15.88:0/802168127 >> 172.21.15.88:0/802168127 conn(0x7fdf3c0f1c70 msgr2=0x7fdf3c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:34.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:34.803+0000 7fdf41462700 1 -- 172.21.15.88:0/802168127 shutdown_connections 2024-02-19T22:15:34.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:34.803+0000 7fdf41462700 1 -- 172.21.15.88:0/802168127 wait complete. 2024-02-19T22:15:34.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:34.803+0000 7fdf41462700 1 Processor -- start 2024-02-19T22:15:34.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:34.803+0000 7fdf41462700 1 -- start start 2024-02-19T22:15:34.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:34.803+0000 7fdf41462700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf3c0f6510 0x7fdf3c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:34.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:34.803+0000 7fdf41462700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3c106cd0 con 0x7fdf3c0f6510 2024-02-19T22:15:34.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:34.804+0000 7fdf3affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf3c0f6510 0x7fdf3c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:34.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:34.804+0000 7fdf3affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf3c0f6510 0x7fdf3c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:36766/0 (socket says 172.21.15.88:36766) 2024-02-19T22:15:34.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:34.804+0000 7fdf3affd700 1 -- 172.21.15.88:0/1026382989 learned_addr learned my addr 172.21.15.88:0/1026382989 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:15:34.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:34.804+0000 7fdf3affd700 1 -- 172.21.15.88:0/1026382989 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf2c005040 con 0x7fdf3c0f6510 2024-02-19T22:15:34.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:34.804+0000 7fdf3affd700 1 --2- 172.21.15.88:0/1026382989 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf3c0f6510 0x7fdf3c1067e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fdf2c005010 tx=0x7fdf2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:34.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:34.804+0000 7fdf237fe700 1 -- 172.21.15.88:0/1026382989 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf2c009320 con 0x7fdf3c0f6510 2024-02-19T22:15:34.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:34.804+0000 7fdf41462700 1 -- 172.21.15.88:0/1026382989 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf3c106ed0 con 0x7fdf3c0f6510 2024-02-19T22:15:34.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:34.804+0000 7fdf237fe700 1 -- 172.21.15.88:0/1026382989 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdf2c013380 con 0x7fdf3c0f6510 2024-02-19T22:15:34.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:34.804+0000 7fdf237fe700 1 -- 172.21.15.88:0/1026382989 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf2c014e50 con 0x7fdf3c0f6510 2024-02-19T22:15:34.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:34.804+0000 7fdf41462700 1 -- 172.21.15.88:0/1026382989 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf3c107360 con 0x7fdf3c0f6510 2024-02-19T22:15:34.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:34.805+0000 7fdf237fe700 1 -- 172.21.15.88:0/1026382989 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdf2c0263a0 con 0x7fdf3c0f6510 2024-02-19T22:15:34.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:34.805+0000 7fdf237fe700 1 --2- 172.21.15.88:0/1026382989 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdf2402ff10 0x7fdf240323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:34.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:34.805+0000 7fdf237fe700 1 -- 172.21.15.88:0/1026382989 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdf2c041290 con 0x7fdf3c0f6510 2024-02-19T22:15:34.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:34.805+0000 7fdf41462700 1 -- 172.21.15.88:0/1026382989 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf3c059100 con 0x7fdf3c0f6510 2024-02-19T22:15:34.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:34.806+0000 7fdf3a7fc700 1 --2- 172.21.15.88:0/1026382989 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdf2402ff10 0x7fdf240323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:34.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:34.806+0000 7fdf3a7fc700 1 --2- 172.21.15.88:0/1026382989 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdf2402ff10 0x7fdf240323d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fdf300096a0 tx=0x7fdf30006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:34.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:34.810+0000 7fdf237fe700 1 -- 172.21.15.88:0/1026382989 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdf2c012920 con 0x7fdf3c0f6510 2024-02-19T22:15:35.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:35.013+0000 7fdf41462700 1 -- 172.21.15.88:0/1026382989 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdf3c0fc0a0 con 0x7fdf3c0f6510 2024-02-19T22:15:35.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:35.014+0000 7fdf237fe700 1 -- 172.21.15.88:0/1026382989 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdf2c02b030 con 0x7fdf3c0f6510 2024-02-19T22:15:35.015 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:15:35.016 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:35.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:35.015+0000 7fdf41462700 1 -- 172.21.15.88:0/1026382989 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdf2402ff10 msgr2=0x7fdf240323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:35.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:35.015+0000 7fdf41462700 1 --2- 172.21.15.88:0/1026382989 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdf2402ff10 0x7fdf240323d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fdf300096a0 tx=0x7fdf30006b40).stop 2024-02-19T22:15:35.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:35.016+0000 7fdf41462700 1 -- 172.21.15.88:0/1026382989 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf3c0f6510 msgr2=0x7fdf3c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:35.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:35.016+0000 7fdf41462700 1 --2- 172.21.15.88:0/1026382989 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf3c0f6510 0x7fdf3c1067e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fdf2c005010 tx=0x7fdf2c00b040).stop 2024-02-19T22:15:35.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:35.016+0000 7fdf41462700 1 -- 172.21.15.88:0/1026382989 shutdown_connections 2024-02-19T22:15:35.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:35.016+0000 7fdf41462700 1 --2- 172.21.15.88:0/1026382989 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdf2402ff10 0x7fdf240323d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:35.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:35.016+0000 7fdf41462700 1 --2- 172.21.15.88:0/1026382989 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf3c0f6510 0x7fdf3c1067e0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:35.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:35.016+0000 7fdf41462700 1 -- 172.21.15.88:0/1026382989 >> 172.21.15.88:0/1026382989 conn(0x7fdf3c0f1c70 msgr2=0x7fdf3c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:35.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:35.016+0000 7fdf41462700 1 -- 172.21.15.88:0/1026382989 shutdown_connections 2024-02-19T22:15:35.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:35.016+0000 7fdf41462700 1 -- 172.21.15.88:0/1026382989 wait complete. 2024-02-19T22:15:35.018 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:15:35.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:35 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:15:33.671240+0000 mgr.smithi052.ghqrtk (mgr.14162) 181 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:35.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:35 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:15:35.014577+0000 mon.smithi052 (mon.0) 221 : audit [DBG] from='client.? 172.21.15.88:0/1026382989' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:36.557 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:36.557 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:37.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:37 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:15:35.671576+0000 mgr.smithi052.ghqrtk (mgr.14162) 182 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:38.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:38.470+0000 7f14d3a99700 1 -- 172.21.15.88:0/859594129 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f14cc0f5410 msgr2=0x7f14cc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:38.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:38.470+0000 7f14d3a99700 1 --2- 172.21.15.88:0/859594129 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f14cc0f5410 0x7f14cc0f5830 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f14c8009fb0 tx=0x7f14c8009b30).stop 2024-02-19T22:15:38.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:38.470+0000 7f14d3a99700 1 -- 172.21.15.88:0/859594129 shutdown_connections 2024-02-19T22:15:38.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:38.470+0000 7f14d3a99700 1 --2- 172.21.15.88:0/859594129 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f14cc0f5410 0x7f14cc0f5830 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:38.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:38.470+0000 7f14d3a99700 1 -- 172.21.15.88:0/859594129 >> 172.21.15.88:0/859594129 conn(0x7f14cc0f0b70 msgr2=0x7f14cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:38.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:38.470+0000 7f14d3a99700 1 -- 172.21.15.88:0/859594129 shutdown_connections 2024-02-19T22:15:38.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:38.470+0000 7f14d3a99700 1 -- 172.21.15.88:0/859594129 wait complete. 2024-02-19T22:15:38.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:38.470+0000 7f14d3a99700 1 Processor -- start 2024-02-19T22:15:38.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:38.471+0000 7f14d3a99700 1 -- start start 2024-02-19T22:15:38.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:38.471+0000 7f14d3a99700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f14cc0f5410 0x7f14cc1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:38.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:38.471+0000 7f14d3a99700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14cc105bd0 con 0x7f14cc0f5410 2024-02-19T22:15:38.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:38.471+0000 7f14d1835700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f14cc0f5410 0x7f14cc1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:38.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:38.471+0000 7f14d1835700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f14cc0f5410 0x7f14cc1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:36776/0 (socket says 172.21.15.88:36776) 2024-02-19T22:15:38.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:38.471+0000 7f14d1835700 1 -- 172.21.15.88:0/2671363872 learned_addr learned my addr 172.21.15.88:0/2671363872 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:15:38.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:38.471+0000 7f14d1835700 1 -- 172.21.15.88:0/2671363872 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c8005040 con 0x7f14cc0f5410 2024-02-19T22:15:38.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:38.471+0000 7f14d1835700 1 --2- 172.21.15.88:0/2671363872 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f14cc0f5410 0x7f14cc1056e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f14c8005010 tx=0x7f14c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:38.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:38.471+0000 7f14c27fc700 1 -- 172.21.15.88:0/2671363872 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14c8009320 con 0x7f14cc0f5410 2024-02-19T22:15:38.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:38.472+0000 7f14c27fc700 1 -- 172.21.15.88:0/2671363872 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f14c8013380 con 0x7f14cc0f5410 2024-02-19T22:15:38.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:38.472+0000 7f14d3a99700 1 -- 172.21.15.88:0/2671363872 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14cc105dd0 con 0x7f14cc0f5410 2024-02-19T22:15:38.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:38.472+0000 7f14c27fc700 1 -- 172.21.15.88:0/2671363872 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14c8014e50 con 0x7f14cc0f5410 2024-02-19T22:15:38.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:38.472+0000 7f14d3a99700 1 -- 172.21.15.88:0/2671363872 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14cc106260 con 0x7f14cc0f5410 2024-02-19T22:15:38.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:38.472+0000 7f14c27fc700 1 -- 172.21.15.88:0/2671363872 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f14c80095f0 con 0x7f14cc0f5410 2024-02-19T22:15:38.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:38.472+0000 7f14d3a99700 1 -- 172.21.15.88:0/2671363872 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14cc045ad0 con 0x7f14cc0f5410 2024-02-19T22:15:38.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:38.472+0000 7f14c27fc700 1 --2- 172.21.15.88:0/2671363872 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f14b802ff10 0x7f14b80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:38.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:38.472+0000 7f14c27fc700 1 -- 172.21.15.88:0/2671363872 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f14c8040fa0 con 0x7f14cc0f5410 2024-02-19T22:15:38.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:38.473+0000 7f14d1034700 1 --2- 172.21.15.88:0/2671363872 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f14b802ff10 0x7f14b80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:38.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:38.473+0000 7f14d1034700 1 --2- 172.21.15.88:0/2671363872 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f14b802ff10 0x7f14b80323d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f14bc0096a0 tx=0x7f14bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:38.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:38.477+0000 7f14c27fc700 1 -- 172.21.15.88:0/2671363872 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f14c8014420 con 0x7f14cc0f5410 2024-02-19T22:15:38.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:38.683+0000 7f14d3a99700 1 -- 172.21.15.88:0/2671363872 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f14cc107110 con 0x7f14cc0f5410 2024-02-19T22:15:38.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:38.683+0000 7f14c27fc700 1 -- 172.21.15.88:0/2671363872 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f14c8010230 con 0x7f14cc0f5410 2024-02-19T22:15:38.684 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:15:38.684 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:38.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:38.685+0000 7f14d3a99700 1 -- 172.21.15.88:0/2671363872 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f14b802ff10 msgr2=0x7f14b80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:38.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:38.685+0000 7f14d3a99700 1 --2- 172.21.15.88:0/2671363872 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f14b802ff10 0x7f14b80323d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f14bc0096a0 tx=0x7f14bc006b40).stop 2024-02-19T22:15:38.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:38.685+0000 7f14d3a99700 1 -- 172.21.15.88:0/2671363872 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f14cc0f5410 msgr2=0x7f14cc1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:38.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:38.685+0000 7f14d3a99700 1 --2- 172.21.15.88:0/2671363872 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f14cc0f5410 0x7f14cc1056e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f14c8005010 tx=0x7f14c800b040).stop 2024-02-19T22:15:38.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:38.685+0000 7f14d3a99700 1 -- 172.21.15.88:0/2671363872 shutdown_connections 2024-02-19T22:15:38.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:38.685+0000 7f14d3a99700 1 --2- 172.21.15.88:0/2671363872 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f14b802ff10 0x7f14b80323d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:38.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:38.685+0000 7f14d3a99700 1 --2- 172.21.15.88:0/2671363872 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f14cc0f5410 0x7f14cc1056e0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:38.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:38.685+0000 7f14d3a99700 1 -- 172.21.15.88:0/2671363872 >> 172.21.15.88:0/2671363872 conn(0x7f14cc0f0b70 msgr2=0x7f14cc0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:38.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:38.685+0000 7f14d3a99700 1 -- 172.21.15.88:0/2671363872 shutdown_connections 2024-02-19T22:15:38.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:38.685+0000 7f14d3a99700 1 -- 172.21.15.88:0/2671363872 wait complete. 2024-02-19T22:15:38.687 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:15:39.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:15:37.671928+0000 mgr.smithi052.ghqrtk (mgr.14162) 183 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:39.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:15:38.683661+0000 mon.smithi052 (mon.0) 222 : audit [DBG] from='client.? 172.21.15.88:0/2671363872' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:40.386 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:40.386 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:41.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:41 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:15:39.672305+0000 mgr.smithi052.ghqrtk (mgr.14162) 184 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:42.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:42.289+0000 7fcb99609700 1 -- 172.21.15.88:0/209844758 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb940f51a0 msgr2=0x7fcb940f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:42.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:42.289+0000 7fcb99609700 1 --2- 172.21.15.88:0/209844758 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb940f51a0 0x7fcb940f55c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7fcb84009fb0 tx=0x7fcb84009b30).stop 2024-02-19T22:15:42.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:42.289+0000 7fcb99609700 1 -- 172.21.15.88:0/209844758 shutdown_connections 2024-02-19T22:15:42.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:42.289+0000 7fcb99609700 1 --2- 172.21.15.88:0/209844758 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb940f51a0 0x7fcb940f55c0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:42.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:42.289+0000 7fcb99609700 1 -- 172.21.15.88:0/209844758 >> 172.21.15.88:0/209844758 conn(0x7fcb940f09a0 msgr2=0x7fcb940f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:42.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:42.289+0000 7fcb99609700 1 -- 172.21.15.88:0/209844758 shutdown_connections 2024-02-19T22:15:42.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:42.289+0000 7fcb99609700 1 -- 172.21.15.88:0/209844758 wait complete. 2024-02-19T22:15:42.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:42.290+0000 7fcb99609700 1 Processor -- start 2024-02-19T22:15:42.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:42.290+0000 7fcb99609700 1 -- start start 2024-02-19T22:15:42.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:42.290+0000 7fcb99609700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb940f51a0 0x7fcb940fd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:42.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:42.290+0000 7fcb99609700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb940fb6f0 con 0x7fcb940f51a0 2024-02-19T22:15:42.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:42.290+0000 7fcb93fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb940f51a0 0x7fcb940fd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:42.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:42.290+0000 7fcb93fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb940f51a0 0x7fcb940fd270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:36790/0 (socket says 172.21.15.88:36790) 2024-02-19T22:15:42.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:42.290+0000 7fcb93fff700 1 -- 172.21.15.88:0/336423918 learned_addr learned my addr 172.21.15.88:0/336423918 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:15:42.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:42.291+0000 7fcb93fff700 1 -- 172.21.15.88:0/336423918 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb84005040 con 0x7fcb940f51a0 2024-02-19T22:15:42.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:42.291+0000 7fcb93fff700 1 --2- 172.21.15.88:0/336423918 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb940f51a0 0x7fcb940fd270 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fcb84009f80 tx=0x7fcb8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:42.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:42.291+0000 7fcb90ff9700 1 -- 172.21.15.88:0/336423918 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb840095f0 con 0x7fcb940f51a0 2024-02-19T22:15:42.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:42.291+0000 7fcb90ff9700 1 -- 172.21.15.88:0/336423918 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcb84007e90 con 0x7fcb940f51a0 2024-02-19T22:15:42.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:42.291+0000 7fcb99609700 1 -- 172.21.15.88:0/336423918 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb940fb890 con 0x7fcb940f51a0 2024-02-19T22:15:42.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:42.291+0000 7fcb90ff9700 1 -- 172.21.15.88:0/336423918 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb84014d80 con 0x7fcb940f51a0 2024-02-19T22:15:42.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:42.291+0000 7fcb99609700 1 -- 172.21.15.88:0/336423918 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb940fbd20 con 0x7fcb940f51a0 2024-02-19T22:15:42.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:42.292+0000 7fcb90ff9700 1 -- 172.21.15.88:0/336423918 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcb84025070 con 0x7fcb940f51a0 2024-02-19T22:15:42.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:42.292+0000 7fcb99609700 1 -- 172.21.15.88:0/336423918 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb94059100 con 0x7fcb940f51a0 2024-02-19T22:15:42.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:42.292+0000 7fcb90ff9700 1 --2- 172.21.15.88:0/336423918 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fcb7c02ff60 0x7fcb7c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:42.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:42.292+0000 7fcb90ff9700 1 -- 172.21.15.88:0/336423918 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcb84040830 con 0x7fcb940f51a0 2024-02-19T22:15:42.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:42.292+0000 7fcb937fe700 1 --2- 172.21.15.88:0/336423918 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fcb7c02ff60 0x7fcb7c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:42.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:42.292+0000 7fcb937fe700 1 --2- 172.21.15.88:0/336423918 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fcb7c02ff60 0x7fcb7c032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fcb880096a0 tx=0x7fcb88006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:42.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:42.297+0000 7fcb90ff9700 1 -- 172.21.15.88:0/336423918 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcb8400f4e0 con 0x7fcb940f51a0 2024-02-19T22:15:42.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:42.500+0000 7fcb99609700 1 -- 172.21.15.88:0/336423918 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcb940fad30 con 0x7fcb940f51a0 2024-02-19T22:15:42.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:42.500+0000 7fcb90ff9700 1 -- 172.21.15.88:0/336423918 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcb8400f300 con 0x7fcb940f51a0 2024-02-19T22:15:42.503 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:15:42.503 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:42.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:42.502+0000 7fcb99609700 1 -- 172.21.15.88:0/336423918 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fcb7c02ff60 msgr2=0x7fcb7c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:42.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:42.502+0000 7fcb99609700 1 --2- 172.21.15.88:0/336423918 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fcb7c02ff60 0x7fcb7c032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fcb880096a0 tx=0x7fcb88006b40).stop 2024-02-19T22:15:42.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:42.502+0000 7fcb99609700 1 -- 172.21.15.88:0/336423918 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb940f51a0 msgr2=0x7fcb940fd270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:42.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:42.502+0000 7fcb99609700 1 --2- 172.21.15.88:0/336423918 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb940f51a0 0x7fcb940fd270 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fcb84009f80 tx=0x7fcb8400b040).stop 2024-02-19T22:15:42.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:42.502+0000 7fcb99609700 1 -- 172.21.15.88:0/336423918 shutdown_connections 2024-02-19T22:15:42.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:42.502+0000 7fcb99609700 1 --2- 172.21.15.88:0/336423918 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fcb7c02ff60 0x7fcb7c032420 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:42.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:42.502+0000 7fcb99609700 1 --2- 172.21.15.88:0/336423918 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb940f51a0 0x7fcb940fd270 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:42.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:42.502+0000 7fcb99609700 1 -- 172.21.15.88:0/336423918 >> 172.21.15.88:0/336423918 conn(0x7fcb940f09a0 msgr2=0x7fcb9405fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:42.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:42.502+0000 7fcb99609700 1 -- 172.21.15.88:0/336423918 shutdown_connections 2024-02-19T22:15:42.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:42.502+0000 7fcb99609700 1 -- 172.21.15.88:0/336423918 wait complete. 2024-02-19T22:15:42.505 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:15:43.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:43 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:15:41.672673+0000 mgr.smithi052.ghqrtk (mgr.14162) 185 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:43.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:43 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:15:42.501913+0000 mon.smithi052 (mon.0) 223 : audit [DBG] from='client.? 172.21.15.88:0/336423918' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:44.159 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:44.159 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:44.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:44 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:15:43.750525+0000 mon.smithi052 (mon.0) 224 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi052.ghqrtk/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:15:44.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:44 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:15:43.753496+0000 mon.smithi052 (mon.0) 225 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi052.ghqrtk/trash_purge_schedule"}]: dispatch 2024-02-19T22:15:45.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:45 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:15:43.673053+0000 mgr.smithi052.ghqrtk (mgr.14162) 186 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:46.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:46.104+0000 7f950080b700 1 -- 172.21.15.88:0/2736793414 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94fc0f51a0 msgr2=0x7f94fc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:46.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:46.104+0000 7f950080b700 1 --2- 172.21.15.88:0/2736793414 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94fc0f51a0 0x7f94fc0f55c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f94ec009fb0 tx=0x7f94ec009b30).stop 2024-02-19T22:15:46.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:46.104+0000 7f950080b700 1 -- 172.21.15.88:0/2736793414 shutdown_connections 2024-02-19T22:15:46.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:46.104+0000 7f950080b700 1 --2- 172.21.15.88:0/2736793414 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94fc0f51a0 0x7f94fc0f55c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:46.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:46.104+0000 7f950080b700 1 -- 172.21.15.88:0/2736793414 >> 172.21.15.88:0/2736793414 conn(0x7f94fc0f09a0 msgr2=0x7f94fc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:46.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:46.104+0000 7f950080b700 1 -- 172.21.15.88:0/2736793414 shutdown_connections 2024-02-19T22:15:46.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:46.104+0000 7f950080b700 1 -- 172.21.15.88:0/2736793414 wait complete. 2024-02-19T22:15:46.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:46.104+0000 7f950080b700 1 Processor -- start 2024-02-19T22:15:46.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:46.104+0000 7f950080b700 1 -- start start 2024-02-19T22:15:46.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:46.105+0000 7f950080b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94fc0f51a0 0x7f94fc109290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:46.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:46.105+0000 7f950080b700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94fc060280 con 0x7f94fc0f51a0 2024-02-19T22:15:46.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:46.105+0000 7f94fad9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94fc0f51a0 0x7f94fc109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:46.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:46.105+0000 7f94fad9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94fc0f51a0 0x7f94fc109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:47484/0 (socket says 172.21.15.88:47484) 2024-02-19T22:15:46.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:46.105+0000 7f94fad9d700 1 -- 172.21.15.88:0/2313865765 learned_addr learned my addr 172.21.15.88:0/2313865765 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:15:46.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:46.105+0000 7f94fad9d700 1 -- 172.21.15.88:0/2313865765 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94ec005040 con 0x7f94fc0f51a0 2024-02-19T22:15:46.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:46.105+0000 7f94fad9d700 1 --2- 172.21.15.88:0/2313865765 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94fc0f51a0 0x7f94fc109290 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f94ec0089f0 tx=0x7f94ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:46.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:46.105+0000 7f94e37fe700 1 -- 172.21.15.88:0/2313865765 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94ec009320 con 0x7f94fc0f51a0 2024-02-19T22:15:46.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:46.106+0000 7f950080b700 1 -- 172.21.15.88:0/2313865765 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94fc060480 con 0x7f94fc0f51a0 2024-02-19T22:15:46.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:46.106+0000 7f94e37fe700 1 -- 172.21.15.88:0/2313865765 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f94ec013380 con 0x7f94fc0f51a0 2024-02-19T22:15:46.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:46.106+0000 7f94e37fe700 1 -- 172.21.15.88:0/2313865765 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94ec014e20 con 0x7f94fc0f51a0 2024-02-19T22:15:46.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:46.106+0000 7f950080b700 1 -- 172.21.15.88:0/2313865765 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94fc1099c0 con 0x7f94fc0f51a0 2024-02-19T22:15:46.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:46.106+0000 7f94e37fe700 1 -- 172.21.15.88:0/2313865765 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f94ec0263a0 con 0x7f94fc0f51a0 2024-02-19T22:15:46.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:46.107+0000 7f950080b700 1 -- 172.21.15.88:0/2313865765 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94fc045ad0 con 0x7f94fc0f51a0 2024-02-19T22:15:46.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:46.107+0000 7f94e37fe700 1 --2- 172.21.15.88:0/2313865765 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f94e402ff10 0x7f94e40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:46.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:46.107+0000 7f94e37fe700 1 -- 172.21.15.88:0/2313865765 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f94ec040ec0 con 0x7f94fc0f51a0 2024-02-19T22:15:46.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:46.107+0000 7f94fa59c700 1 --2- 172.21.15.88:0/2313865765 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f94e402ff10 0x7f94e40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:46.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:46.107+0000 7f94fa59c700 1 --2- 172.21.15.88:0/2313865765 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f94e402ff10 0x7f94e40323d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f94f00096a0 tx=0x7f94f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:46.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:46.112+0000 7f94e37fe700 1 -- 172.21.15.88:0/2313865765 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f94ec01dde0 con 0x7f94fc0f51a0 2024-02-19T22:15:46.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:46.317+0000 7f950080b700 1 -- 172.21.15.88:0/2313865765 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f94fc0fad30 con 0x7f94fc0f51a0 2024-02-19T22:15:46.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:46.317+0000 7f94e37fe700 1 -- 172.21.15.88:0/2313865765 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f94ec010230 con 0x7f94fc0f51a0 2024-02-19T22:15:46.319 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:15:46.320 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:46.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:46.319+0000 7f950080b700 1 -- 172.21.15.88:0/2313865765 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f94e402ff10 msgr2=0x7f94e40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:46.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:46.319+0000 7f950080b700 1 --2- 172.21.15.88:0/2313865765 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f94e402ff10 0x7f94e40323d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f94f00096a0 tx=0x7f94f0006b40).stop 2024-02-19T22:15:46.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:46.319+0000 7f950080b700 1 -- 172.21.15.88:0/2313865765 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94fc0f51a0 msgr2=0x7f94fc109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:46.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:46.319+0000 7f950080b700 1 --2- 172.21.15.88:0/2313865765 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94fc0f51a0 0x7f94fc109290 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f94ec0089f0 tx=0x7f94ec00b040).stop 2024-02-19T22:15:46.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:46.319+0000 7f950080b700 1 -- 172.21.15.88:0/2313865765 shutdown_connections 2024-02-19T22:15:46.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:46.319+0000 7f950080b700 1 --2- 172.21.15.88:0/2313865765 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f94e402ff10 0x7f94e40323d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:46.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:46.319+0000 7f950080b700 1 --2- 172.21.15.88:0/2313865765 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94fc0f51a0 0x7f94fc109290 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:46.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:46.319+0000 7f950080b700 1 -- 172.21.15.88:0/2313865765 >> 172.21.15.88:0/2313865765 conn(0x7f94fc0f09a0 msgr2=0x7f94fc0f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:46.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:46.319+0000 7f950080b700 1 -- 172.21.15.88:0/2313865765 shutdown_connections 2024-02-19T22:15:46.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:46.319+0000 7f950080b700 1 -- 172.21.15.88:0/2313865765 wait complete. 2024-02-19T22:15:46.322 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:15:46.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:15:46.318806+0000 mon.smithi052 (mon.0) 226 : audit [DBG] from='client.? 172.21.15.88:0/2313865765' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:47.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:47 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:15:45.673442+0000 mgr.smithi052.ghqrtk (mgr.14162) 187 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:47.948 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:47.948 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:49.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:49 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:15:47.673894+0000 mgr.smithi052.ghqrtk (mgr.14162) 188 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:49.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:49.828+0000 7f3573e50700 1 -- 172.21.15.88:0/4272541547 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f356c060280 msgr2=0x7f356c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:49.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:49.828+0000 7f3573e50700 1 --2- 172.21.15.88:0/4272541547 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f356c060280 0x7f356c0f90f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f355c009fb0 tx=0x7f355c009b30).stop 2024-02-19T22:15:49.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:49.828+0000 7f3573e50700 1 -- 172.21.15.88:0/4272541547 shutdown_connections 2024-02-19T22:15:49.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:49.828+0000 7f3573e50700 1 --2- 172.21.15.88:0/4272541547 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f356c060280 0x7f356c0f90f0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:49.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:49.828+0000 7f3573e50700 1 -- 172.21.15.88:0/4272541547 >> 172.21.15.88:0/4272541547 conn(0x7f356c0f0b70 msgr2=0x7f356c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:49.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:49.828+0000 7f3573e50700 1 -- 172.21.15.88:0/4272541547 shutdown_connections 2024-02-19T22:15:49.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:49.828+0000 7f3573e50700 1 -- 172.21.15.88:0/4272541547 wait complete. 2024-02-19T22:15:49.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:49.829+0000 7f3573e50700 1 Processor -- start 2024-02-19T22:15:49.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:49.829+0000 7f3573e50700 1 -- start start 2024-02-19T22:15:49.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:49.829+0000 7f3573e50700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f356c060280 0x7f356c106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:49.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:49.829+0000 7f3573e50700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f356c106c70 con 0x7f356c060280 2024-02-19T22:15:49.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:49.829+0000 7f3571bec700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f356c060280 0x7f356c106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:49.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:49.829+0000 7f3571bec700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f356c060280 0x7f356c106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:47500/0 (socket says 172.21.15.88:47500) 2024-02-19T22:15:49.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:49.829+0000 7f3571bec700 1 -- 172.21.15.88:0/2507741486 learned_addr learned my addr 172.21.15.88:0/2507741486 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:15:49.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:49.829+0000 7f3571bec700 1 -- 172.21.15.88:0/2507741486 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f355c005040 con 0x7f356c060280 2024-02-19T22:15:49.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:49.829+0000 7f3571bec700 1 --2- 172.21.15.88:0/2507741486 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f356c060280 0x7f356c106780 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f355c0089f0 tx=0x7f355c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:49.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:49.830+0000 7f35627fc700 1 -- 172.21.15.88:0/2507741486 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f355c009320 con 0x7f356c060280 2024-02-19T22:15:49.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:49.830+0000 7f3573e50700 1 -- 172.21.15.88:0/2507741486 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f356c106e70 con 0x7f356c060280 2024-02-19T22:15:49.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:49.830+0000 7f35627fc700 1 -- 172.21.15.88:0/2507741486 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f355c013380 con 0x7f356c060280 2024-02-19T22:15:49.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:49.830+0000 7f35627fc700 1 -- 172.21.15.88:0/2507741486 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f355c014e20 con 0x7f356c060280 2024-02-19T22:15:49.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:49.830+0000 7f3573e50700 1 -- 172.21.15.88:0/2507741486 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f356c107300 con 0x7f356c060280 2024-02-19T22:15:49.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:49.830+0000 7f35627fc700 1 -- 172.21.15.88:0/2507741486 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f355c014420 con 0x7f356c060280 2024-02-19T22:15:49.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:49.830+0000 7f3573e50700 1 -- 172.21.15.88:0/2507741486 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f356c045ad0 con 0x7f356c060280 2024-02-19T22:15:49.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:49.831+0000 7f35627fc700 1 --2- 172.21.15.88:0/2507741486 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f355802ff10 0x7f35580323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:49.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:49.831+0000 7f35627fc700 1 -- 172.21.15.88:0/2507741486 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f355c040f00 con 0x7f356c060280 2024-02-19T22:15:49.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:49.831+0000 7f35713eb700 1 --2- 172.21.15.88:0/2507741486 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f355802ff10 0x7f35580323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:49.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:49.831+0000 7f35713eb700 1 --2- 172.21.15.88:0/2507741486 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f355802ff10 0x7f35580323d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f35680096a0 tx=0x7f3568006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:49.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:49.836+0000 7f35627fc700 1 -- 172.21.15.88:0/2507741486 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f355c0095f0 con 0x7f356c060280 2024-02-19T22:15:50.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:50.040+0000 7f3573e50700 1 -- 172.21.15.88:0/2507741486 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f356c1081b0 con 0x7f356c060280 2024-02-19T22:15:50.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:50.041+0000 7f35627fc700 1 -- 172.21.15.88:0/2507741486 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f355c01db70 con 0x7f356c060280 2024-02-19T22:15:50.043 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:15:50.043 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:50.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:50.042+0000 7f3573e50700 1 -- 172.21.15.88:0/2507741486 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f355802ff10 msgr2=0x7f35580323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:50.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:50.042+0000 7f3573e50700 1 --2- 172.21.15.88:0/2507741486 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f355802ff10 0x7f35580323d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f35680096a0 tx=0x7f3568006b40).stop 2024-02-19T22:15:50.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:50.042+0000 7f3573e50700 1 -- 172.21.15.88:0/2507741486 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f356c060280 msgr2=0x7f356c106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:50.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:50.042+0000 7f3573e50700 1 --2- 172.21.15.88:0/2507741486 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f356c060280 0x7f356c106780 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f355c0089f0 tx=0x7f355c00b040).stop 2024-02-19T22:15:50.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:50.042+0000 7f3573e50700 1 -- 172.21.15.88:0/2507741486 shutdown_connections 2024-02-19T22:15:50.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:50.042+0000 7f3573e50700 1 --2- 172.21.15.88:0/2507741486 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f355802ff10 0x7f35580323d0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:50.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:50.042+0000 7f3573e50700 1 --2- 172.21.15.88:0/2507741486 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f356c060280 0x7f356c106780 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:50.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:50.042+0000 7f3573e50700 1 -- 172.21.15.88:0/2507741486 >> 172.21.15.88:0/2507741486 conn(0x7f356c0f0b70 msgr2=0x7f356c0f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:50.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:50.042+0000 7f3573e50700 1 -- 172.21.15.88:0/2507741486 shutdown_connections 2024-02-19T22:15:50.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:50.042+0000 7f3573e50700 1 -- 172.21.15.88:0/2507741486 wait complete. 2024-02-19T22:15:50.045 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:15:50.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:50 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:15:50.042050+0000 mon.smithi052 (mon.0) 227 : audit [DBG] from='client.? 172.21.15.88:0/2507741486' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:51.658 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:51.658 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:51.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:51 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:15:49.674286+0000 mgr.smithi052.ghqrtk (mgr.14162) 189 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:53.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:53.498+0000 7fa06626b700 1 -- 172.21.15.88:0/2352741720 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa060060280 msgr2=0x7fa0600f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:53.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:53.498+0000 7fa06626b700 1 --2- 172.21.15.88:0/2352741720 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa060060280 0x7fa0600f90f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7fa048009fb0 tx=0x7fa048009b30).stop 2024-02-19T22:15:53.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:53.498+0000 7fa06626b700 1 -- 172.21.15.88:0/2352741720 shutdown_connections 2024-02-19T22:15:53.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:53.498+0000 7fa06626b700 1 --2- 172.21.15.88:0/2352741720 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa060060280 0x7fa0600f90f0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:53.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:53.498+0000 7fa06626b700 1 -- 172.21.15.88:0/2352741720 >> 172.21.15.88:0/2352741720 conn(0x7fa0600f0b70 msgr2=0x7fa0600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:53.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:53.498+0000 7fa06626b700 1 -- 172.21.15.88:0/2352741720 shutdown_connections 2024-02-19T22:15:53.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:53.498+0000 7fa06626b700 1 -- 172.21.15.88:0/2352741720 wait complete. 2024-02-19T22:15:53.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:53.498+0000 7fa06626b700 1 Processor -- start 2024-02-19T22:15:53.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:53.499+0000 7fa06626b700 1 -- start start 2024-02-19T22:15:53.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:53.499+0000 7fa06626b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa060060280 0x7fa060106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:53.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:53.499+0000 7fa06626b700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa060106c70 con 0x7fa060060280 2024-02-19T22:15:53.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:53.499+0000 7fa05ffff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa060060280 0x7fa060106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:53.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:53.499+0000 7fa05ffff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa060060280 0x7fa060106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:49680/0 (socket says 172.21.15.88:49680) 2024-02-19T22:15:53.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:53.499+0000 7fa05ffff700 1 -- 172.21.15.88:0/3099293258 learned_addr learned my addr 172.21.15.88:0/3099293258 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:15:53.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:53.499+0000 7fa05ffff700 1 -- 172.21.15.88:0/3099293258 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa048005040 con 0x7fa060060280 2024-02-19T22:15:53.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:53.500+0000 7fa05ffff700 1 --2- 172.21.15.88:0/3099293258 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa060060280 0x7fa060106780 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7fa048009830 tx=0x7fa04800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:53.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:53.500+0000 7fa05cff9700 1 -- 172.21.15.88:0/3099293258 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa048009320 con 0x7fa060060280 2024-02-19T22:15:53.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:53.500+0000 7fa06626b700 1 -- 172.21.15.88:0/3099293258 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa060106e70 con 0x7fa060060280 2024-02-19T22:15:53.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:53.500+0000 7fa05cff9700 1 -- 172.21.15.88:0/3099293258 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa048013380 con 0x7fa060060280 2024-02-19T22:15:53.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:53.500+0000 7fa05cff9700 1 -- 172.21.15.88:0/3099293258 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa048014420 con 0x7fa060060280 2024-02-19T22:15:53.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:53.500+0000 7fa06626b700 1 -- 172.21.15.88:0/3099293258 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa060107300 con 0x7fa060060280 2024-02-19T22:15:53.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:53.501+0000 7fa05cff9700 1 -- 172.21.15.88:0/3099293258 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa04801e420 con 0x7fa060060280 2024-02-19T22:15:53.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:53.501+0000 7fa06626b700 1 -- 172.21.15.88:0/3099293258 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa060045ad0 con 0x7fa060060280 2024-02-19T22:15:53.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:53.501+0000 7fa05cff9700 1 --2- 172.21.15.88:0/3099293258 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa04c02ff60 0x7fa04c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:53.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:53.501+0000 7fa05cff9700 1 -- 172.21.15.88:0/3099293258 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa048040eb0 con 0x7fa060060280 2024-02-19T22:15:53.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:53.501+0000 7fa05f7fe700 1 --2- 172.21.15.88:0/3099293258 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa04c02ff60 0x7fa04c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:53.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:53.501+0000 7fa05f7fe700 1 --2- 172.21.15.88:0/3099293258 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa04c02ff60 0x7fa04c032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fa0500096a0 tx=0x7fa050006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:53.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:53.506+0000 7fa05cff9700 1 -- 172.21.15.88:0/3099293258 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa04801cb10 con 0x7fa060060280 2024-02-19T22:15:53.592 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:53 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:15:51.674575+0000 mgr.smithi052.ghqrtk (mgr.14162) 190 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:53.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:53.709+0000 7fa06626b700 1 -- 172.21.15.88:0/3099293258 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa0600fc090 con 0x7fa060060280 2024-02-19T22:15:53.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:53.710+0000 7fa05cff9700 1 -- 172.21.15.88:0/3099293258 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa048004490 con 0x7fa060060280 2024-02-19T22:15:53.712 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:15:53.712 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:53.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:53.711+0000 7fa06626b700 1 -- 172.21.15.88:0/3099293258 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa04c02ff60 msgr2=0x7fa04c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:53.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:53.711+0000 7fa06626b700 1 --2- 172.21.15.88:0/3099293258 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa04c02ff60 0x7fa04c032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fa0500096a0 tx=0x7fa050006b40).stop 2024-02-19T22:15:53.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:53.711+0000 7fa06626b700 1 -- 172.21.15.88:0/3099293258 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa060060280 msgr2=0x7fa060106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:53.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:53.711+0000 7fa06626b700 1 --2- 172.21.15.88:0/3099293258 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa060060280 0x7fa060106780 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7fa048009830 tx=0x7fa04800b040).stop 2024-02-19T22:15:53.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:53.711+0000 7fa06626b700 1 -- 172.21.15.88:0/3099293258 shutdown_connections 2024-02-19T22:15:53.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:53.711+0000 7fa06626b700 1 --2- 172.21.15.88:0/3099293258 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa04c02ff60 0x7fa04c032420 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:53.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:53.711+0000 7fa06626b700 1 --2- 172.21.15.88:0/3099293258 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa060060280 0x7fa060106780 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:53.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:53.711+0000 7fa06626b700 1 -- 172.21.15.88:0/3099293258 >> 172.21.15.88:0/3099293258 conn(0x7fa0600f0b70 msgr2=0x7fa0600f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:53.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:53.711+0000 7fa06626b700 1 -- 172.21.15.88:0/3099293258 shutdown_connections 2024-02-19T22:15:53.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:53.711+0000 7fa06626b700 1 -- 172.21.15.88:0/3099293258 wait complete. 2024-02-19T22:15:53.715 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:15:54.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:54 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:15:53.711362+0000 mon.smithi052 (mon.0) 228 : audit [DBG] from='client.? 172.21.15.88:0/3099293258' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:55.362 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:55.363 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:55.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:55 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:15:53.674863+0000 mgr.smithi052.ghqrtk (mgr.14162) 191 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:57.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:57.233+0000 7fdc06bd9700 1 -- 172.21.15.88:0/35127884 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdc000f6510 msgr2=0x7fdc000f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:57.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:57.233+0000 7fdc06bd9700 1 --2- 172.21.15.88:0/35127884 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdc000f6510 0x7fdc000f6930 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7fdbf4009fb0 tx=0x7fdbf4009b30).stop 2024-02-19T22:15:57.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:57.233+0000 7fdc06bd9700 1 -- 172.21.15.88:0/35127884 shutdown_connections 2024-02-19T22:15:57.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:57.233+0000 7fdc06bd9700 1 --2- 172.21.15.88:0/35127884 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdc000f6510 0x7fdc000f6930 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:57.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:57.233+0000 7fdc06bd9700 1 -- 172.21.15.88:0/35127884 >> 172.21.15.88:0/35127884 conn(0x7fdc000f1c70 msgr2=0x7fdc000f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:57.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:57.233+0000 7fdc06bd9700 1 -- 172.21.15.88:0/35127884 shutdown_connections 2024-02-19T22:15:57.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:57.233+0000 7fdc06bd9700 1 -- 172.21.15.88:0/35127884 wait complete. 2024-02-19T22:15:57.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:57.234+0000 7fdc06bd9700 1 Processor -- start 2024-02-19T22:15:57.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:57.234+0000 7fdc06bd9700 1 -- start start 2024-02-19T22:15:57.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:57.234+0000 7fdc06bd9700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdc000f6510 0x7fdc001067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:57.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:57.234+0000 7fdc06bd9700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc00106cd0 con 0x7fdc000f6510 2024-02-19T22:15:57.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:57.234+0000 7fdc04975700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdc000f6510 0x7fdc001067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:57.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:57.235+0000 7fdc04975700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdc000f6510 0x7fdc001067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:49688/0 (socket says 172.21.15.88:49688) 2024-02-19T22:15:57.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:57.235+0000 7fdc04975700 1 -- 172.21.15.88:0/2256960131 learned_addr learned my addr 172.21.15.88:0/2256960131 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:15:57.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:57.235+0000 7fdc04975700 1 -- 172.21.15.88:0/2256960131 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbf4005040 con 0x7fdc000f6510 2024-02-19T22:15:57.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:57.235+0000 7fdc04975700 1 --2- 172.21.15.88:0/2256960131 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdc000f6510 0x7fdc001067e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7fdbf4005010 tx=0x7fdbf400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:57.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:57.235+0000 7fdbfd7fa700 1 -- 172.21.15.88:0/2256960131 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbf4009320 con 0x7fdc000f6510 2024-02-19T22:15:57.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:57.235+0000 7fdbfd7fa700 1 -- 172.21.15.88:0/2256960131 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdbf4013380 con 0x7fdc000f6510 2024-02-19T22:15:57.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:57.235+0000 7fdc06bd9700 1 -- 172.21.15.88:0/2256960131 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc00106ed0 con 0x7fdc000f6510 2024-02-19T22:15:57.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:57.235+0000 7fdbfd7fa700 1 -- 172.21.15.88:0/2256960131 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbf4014e50 con 0x7fdc000f6510 2024-02-19T22:15:57.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:57.235+0000 7fdc06bd9700 1 -- 172.21.15.88:0/2256960131 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc00107360 con 0x7fdc000f6510 2024-02-19T22:15:57.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:57.236+0000 7fdbfd7fa700 1 -- 172.21.15.88:0/2256960131 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdbf40263a0 con 0x7fdc000f6510 2024-02-19T22:15:57.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:57.236+0000 7fdbfd7fa700 1 --2- 172.21.15.88:0/2256960131 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdbf002ff10 0x7fdbf00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:57.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:57.236+0000 7fdbfd7fa700 1 -- 172.21.15.88:0/2256960131 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdbf4040e80 con 0x7fdc000f6510 2024-02-19T22:15:57.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:57.236+0000 7fdc06bd9700 1 -- 172.21.15.88:0/2256960131 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc00045ad0 con 0x7fdc000f6510 2024-02-19T22:15:57.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:57.236+0000 7fdbfffff700 1 --2- 172.21.15.88:0/2256960131 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdbf002ff10 0x7fdbf00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:57.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:57.237+0000 7fdbfffff700 1 --2- 172.21.15.88:0/2256960131 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdbf002ff10 0x7fdbf00323d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fdbec0096a0 tx=0x7fdbec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:57.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:57.241+0000 7fdbfd7fa700 1 -- 172.21.15.88:0/2256960131 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdbf4012920 con 0x7fdc000f6510 2024-02-19T22:15:57.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:57.443+0000 7fdc06bd9700 1 -- 172.21.15.88:0/2256960131 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdc00108210 con 0x7fdc000f6510 2024-02-19T22:15:57.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:57.444+0000 7fdbfd7fa700 1 -- 172.21.15.88:0/2256960131 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdbf402b020 con 0x7fdc000f6510 2024-02-19T22:15:57.446 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:15:57.446 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:57.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:57.445+0000 7fdc06bd9700 1 -- 172.21.15.88:0/2256960131 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdbf002ff10 msgr2=0x7fdbf00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:57.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:57.445+0000 7fdc06bd9700 1 --2- 172.21.15.88:0/2256960131 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdbf002ff10 0x7fdbf00323d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fdbec0096a0 tx=0x7fdbec006b40).stop 2024-02-19T22:15:57.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:57.445+0000 7fdc06bd9700 1 -- 172.21.15.88:0/2256960131 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdc000f6510 msgr2=0x7fdc001067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:57.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:57.445+0000 7fdc06bd9700 1 --2- 172.21.15.88:0/2256960131 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdc000f6510 0x7fdc001067e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7fdbf4005010 tx=0x7fdbf400b040).stop 2024-02-19T22:15:57.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:57.445+0000 7fdc06bd9700 1 -- 172.21.15.88:0/2256960131 shutdown_connections 2024-02-19T22:15:57.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:57.445+0000 7fdc06bd9700 1 --2- 172.21.15.88:0/2256960131 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdbf002ff10 0x7fdbf00323d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:57.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:57.445+0000 7fdc06bd9700 1 --2- 172.21.15.88:0/2256960131 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdc000f6510 0x7fdc001067e0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:57.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:57.445+0000 7fdc06bd9700 1 -- 172.21.15.88:0/2256960131 >> 172.21.15.88:0/2256960131 conn(0x7fdc000f1c70 msgr2=0x7fdc000f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:57.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:57.446+0000 7fdc06bd9700 1 -- 172.21.15.88:0/2256960131 shutdown_connections 2024-02-19T22:15:57.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:15:57.446+0000 7fdc06bd9700 1 -- 172.21.15.88:0/2256960131 wait complete. 2024-02-19T22:15:57.449 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:15:57.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:57 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:15:55.675182+0000 mgr.smithi052.ghqrtk (mgr.14162) 192 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:58.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:58 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:15:57.445333+0000 mon.smithi052 (mon.0) 229 : audit [DBG] from='client.? 172.21.15.88:0/2256960131' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:59.042 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:59.042 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:59.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:15:59 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:15:57.675561+0000 mgr.smithi052.ghqrtk (mgr.14162) 193 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:00.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:00.829+0000 7f0c0bfff700 1 -- 172.21.15.88:0/2714199875 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0c0c0f51a0 msgr2=0x7f0c0c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:00.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:00.829+0000 7f0c0bfff700 1 --2- 172.21.15.88:0/2714199875 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0c0c0f51a0 0x7f0c0c0f55c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f0c00009fb0 tx=0x7f0c00009b30).stop 2024-02-19T22:16:00.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:00.829+0000 7f0c0bfff700 1 -- 172.21.15.88:0/2714199875 shutdown_connections 2024-02-19T22:16:00.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:00.829+0000 7f0c0bfff700 1 --2- 172.21.15.88:0/2714199875 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0c0c0f51a0 0x7f0c0c0f55c0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:00.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:00.829+0000 7f0c0bfff700 1 -- 172.21.15.88:0/2714199875 >> 172.21.15.88:0/2714199875 conn(0x7f0c0c0f09a0 msgr2=0x7f0c0c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:00.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:00.829+0000 7f0c0bfff700 1 -- 172.21.15.88:0/2714199875 shutdown_connections 2024-02-19T22:16:00.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:00.829+0000 7f0c0bfff700 1 -- 172.21.15.88:0/2714199875 wait complete. 2024-02-19T22:16:00.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:00.829+0000 7f0c0bfff700 1 Processor -- start 2024-02-19T22:16:00.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:00.830+0000 7f0c0bfff700 1 -- start start 2024-02-19T22:16:00.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:00.830+0000 7f0c0bfff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0c0c0f51a0 0x7f0c0c109290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:00.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:00.830+0000 7f0c0bfff700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0c060280 con 0x7f0c0c0f51a0 2024-02-19T22:16:00.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:00.830+0000 7f0c0affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0c0c0f51a0 0x7f0c0c109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:00.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:00.830+0000 7f0c0affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0c0c0f51a0 0x7f0c0c109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:49706/0 (socket says 172.21.15.88:49706) 2024-02-19T22:16:00.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:00.830+0000 7f0c0affd700 1 -- 172.21.15.88:0/1111241137 learned_addr learned my addr 172.21.15.88:0/1111241137 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:16:00.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:00.830+0000 7f0c0affd700 1 -- 172.21.15.88:0/1111241137 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c00005040 con 0x7f0c0c0f51a0 2024-02-19T22:16:00.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:00.830+0000 7f0c0affd700 1 --2- 172.21.15.88:0/1111241137 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0c0c0f51a0 0x7f0c0c109290 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f0c000089f0 tx=0x7f0c0000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:00.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:00.831+0000 7f0bf37fe700 1 -- 172.21.15.88:0/1111241137 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c00009320 con 0x7f0c0c0f51a0 2024-02-19T22:16:00.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:00.831+0000 7f0bf37fe700 1 -- 172.21.15.88:0/1111241137 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0c00013380 con 0x7f0c0c0f51a0 2024-02-19T22:16:00.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:00.831+0000 7f0c0bfff700 1 -- 172.21.15.88:0/1111241137 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c0c060480 con 0x7f0c0c0f51a0 2024-02-19T22:16:00.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:00.831+0000 7f0bf37fe700 1 -- 172.21.15.88:0/1111241137 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c00014e20 con 0x7f0c0c0f51a0 2024-02-19T22:16:00.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:00.831+0000 7f0c0bfff700 1 -- 172.21.15.88:0/1111241137 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c0c1099c0 con 0x7f0c0c0f51a0 2024-02-19T22:16:00.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:00.831+0000 7f0bf37fe700 1 -- 172.21.15.88:0/1111241137 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0c000095f0 con 0x7f0c0c0f51a0 2024-02-19T22:16:00.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:00.831+0000 7f0c0bfff700 1 -- 172.21.15.88:0/1111241137 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c0c045ad0 con 0x7f0c0c0f51a0 2024-02-19T22:16:00.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:00.832+0000 7f0bf37fe700 1 --2- 172.21.15.88:0/1111241137 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f0bf402ff10 0x7f0bf40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:00.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:00.832+0000 7f0bf37fe700 1 -- 172.21.15.88:0/1111241137 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0c00040e90 con 0x7f0c0c0f51a0 2024-02-19T22:16:00.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:00.832+0000 7f0c0a7fc700 1 --2- 172.21.15.88:0/1111241137 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f0bf402ff10 0x7f0bf40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:00.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:00.832+0000 7f0c0a7fc700 1 --2- 172.21.15.88:0/1111241137 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f0bf402ff10 0x7f0bf40323d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f0bfc0096a0 tx=0x7f0bfc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:00.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:00.837+0000 7f0bf37fe700 1 -- 172.21.15.88:0/1111241137 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0c00014420 con 0x7f0c0c0f51a0 2024-02-19T22:16:01.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:01.042+0000 7f0c0bfff700 1 -- 172.21.15.88:0/1111241137 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0c0c0fad30 con 0x7f0c0c0f51a0 2024-02-19T22:16:01.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:01.042+0000 7f0bf37fe700 1 -- 172.21.15.88:0/1111241137 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0c0001d980 con 0x7f0c0c0f51a0 2024-02-19T22:16:01.045 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:16:01.045 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:01.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:01.044+0000 7f0c0bfff700 1 -- 172.21.15.88:0/1111241137 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f0bf402ff10 msgr2=0x7f0bf40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:01.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:01.044+0000 7f0c0bfff700 1 --2- 172.21.15.88:0/1111241137 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f0bf402ff10 0x7f0bf40323d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f0bfc0096a0 tx=0x7f0bfc006b40).stop 2024-02-19T22:16:01.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:01.044+0000 7f0c0bfff700 1 -- 172.21.15.88:0/1111241137 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0c0c0f51a0 msgr2=0x7f0c0c109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:01.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:01.044+0000 7f0c0bfff700 1 --2- 172.21.15.88:0/1111241137 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0c0c0f51a0 0x7f0c0c109290 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f0c000089f0 tx=0x7f0c0000b040).stop 2024-02-19T22:16:01.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:01.044+0000 7f0c0bfff700 1 -- 172.21.15.88:0/1111241137 shutdown_connections 2024-02-19T22:16:01.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:01.044+0000 7f0c0bfff700 1 --2- 172.21.15.88:0/1111241137 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f0bf402ff10 0x7f0bf40323d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:01.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:01.044+0000 7f0c0bfff700 1 --2- 172.21.15.88:0/1111241137 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0c0c0f51a0 0x7f0c0c109290 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:01.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:01.044+0000 7f0c0bfff700 1 -- 172.21.15.88:0/1111241137 >> 172.21.15.88:0/1111241137 conn(0x7f0c0c0f09a0 msgr2=0x7f0c0c0f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:01.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:01.044+0000 7f0c0bfff700 1 -- 172.21.15.88:0/1111241137 shutdown_connections 2024-02-19T22:16:01.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:01.044+0000 7f0c0bfff700 1 -- 172.21.15.88:0/1111241137 wait complete. 2024-02-19T22:16:01.047 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:16:01.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:01 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:15:59.675925+0000 mgr.smithi052.ghqrtk (mgr.14162) 194 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:01.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:01 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:16:01.043921+0000 mon.smithi052 (mon.0) 230 : audit [DBG] from='client.? 172.21.15.88:0/1111241137' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:02.691 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:02.691 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:03.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:03 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:16:01.676203+0000 mgr.smithi052.ghqrtk (mgr.14162) 195 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:04.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:04.591+0000 7fdb76733700 1 -- 172.21.15.88:0/3504520332 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdb70060280 msgr2=0x7fdb700f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:04.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:04.591+0000 7fdb76733700 1 --2- 172.21.15.88:0/3504520332 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdb70060280 0x7fdb700f90f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7fdb60009fb0 tx=0x7fdb60009b30).stop 2024-02-19T22:16:04.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:04.591+0000 7fdb76733700 1 -- 172.21.15.88:0/3504520332 shutdown_connections 2024-02-19T22:16:04.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:04.591+0000 7fdb76733700 1 --2- 172.21.15.88:0/3504520332 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdb70060280 0x7fdb700f90f0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:04.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:04.591+0000 7fdb76733700 1 -- 172.21.15.88:0/3504520332 >> 172.21.15.88:0/3504520332 conn(0x7fdb700f0b70 msgr2=0x7fdb700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:04.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:04.591+0000 7fdb76733700 1 -- 172.21.15.88:0/3504520332 shutdown_connections 2024-02-19T22:16:04.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:04.591+0000 7fdb76733700 1 -- 172.21.15.88:0/3504520332 wait complete. 2024-02-19T22:16:04.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:04.592+0000 7fdb76733700 1 Processor -- start 2024-02-19T22:16:04.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:04.592+0000 7fdb76733700 1 -- start start 2024-02-19T22:16:04.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:04.592+0000 7fdb76733700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdb70060280 0x7fdb70106760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:04.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:04.592+0000 7fdb76733700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb70106c50 con 0x7fdb70060280 2024-02-19T22:16:04.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:04.592+0000 7fdb6ffff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdb70060280 0x7fdb70106760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:04.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:04.592+0000 7fdb6ffff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdb70060280 0x7fdb70106760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:43332/0 (socket says 172.21.15.88:43332) 2024-02-19T22:16:04.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:04.592+0000 7fdb6ffff700 1 -- 172.21.15.88:0/715302863 learned_addr learned my addr 172.21.15.88:0/715302863 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:16:04.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:04.593+0000 7fdb6ffff700 1 -- 172.21.15.88:0/715302863 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb60005040 con 0x7fdb70060280 2024-02-19T22:16:04.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:04.593+0000 7fdb6ffff700 1 --2- 172.21.15.88:0/715302863 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdb70060280 0x7fdb70106760 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7fdb600089f0 tx=0x7fdb6000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:04.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:04.593+0000 7fdb6cff9700 1 -- 172.21.15.88:0/715302863 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb60009320 con 0x7fdb70060280 2024-02-19T22:16:04.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:04.593+0000 7fdb6cff9700 1 -- 172.21.15.88:0/715302863 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdb60013380 con 0x7fdb70060280 2024-02-19T22:16:04.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:04.593+0000 7fdb76733700 1 -- 172.21.15.88:0/715302863 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb70106e50 con 0x7fdb70060280 2024-02-19T22:16:04.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:04.593+0000 7fdb6cff9700 1 -- 172.21.15.88:0/715302863 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb60014e20 con 0x7fdb70060280 2024-02-19T22:16:04.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:04.593+0000 7fdb76733700 1 -- 172.21.15.88:0/715302863 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb701072e0 con 0x7fdb70060280 2024-02-19T22:16:04.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:04.594+0000 7fdb6cff9700 1 -- 172.21.15.88:0/715302863 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdb600095f0 con 0x7fdb70060280 2024-02-19T22:16:04.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:04.594+0000 7fdb76733700 1 -- 172.21.15.88:0/715302863 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb70045ad0 con 0x7fdb70060280 2024-02-19T22:16:04.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:04.594+0000 7fdb6cff9700 1 --2- 172.21.15.88:0/715302863 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdb5802ff10 0x7fdb580323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:04.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:04.594+0000 7fdb6cff9700 1 -- 172.21.15.88:0/715302863 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdb60040f60 con 0x7fdb70060280 2024-02-19T22:16:04.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:04.594+0000 7fdb6f7fe700 1 --2- 172.21.15.88:0/715302863 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdb5802ff10 0x7fdb580323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:04.598 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:04.595+0000 7fdb6f7fe700 1 --2- 172.21.15.88:0/715302863 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdb5802ff10 0x7fdb580323d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fdb640096a0 tx=0x7fdb64006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:04.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:04.599+0000 7fdb6cff9700 1 -- 172.21.15.88:0/715302863 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdb60014420 con 0x7fdb70060280 2024-02-19T22:16:04.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:04.806+0000 7fdb76733700 1 -- 172.21.15.88:0/715302863 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdb70108190 con 0x7fdb70060280 2024-02-19T22:16:04.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:04.808+0000 7fdb6cff9700 1 -- 172.21.15.88:0/715302863 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdb60010230 con 0x7fdb70060280 2024-02-19T22:16:04.809 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:16:04.810 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:04.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:04.809+0000 7fdb76733700 1 -- 172.21.15.88:0/715302863 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdb5802ff10 msgr2=0x7fdb580323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:04.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:04.809+0000 7fdb76733700 1 --2- 172.21.15.88:0/715302863 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdb5802ff10 0x7fdb580323d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fdb640096a0 tx=0x7fdb64006b40).stop 2024-02-19T22:16:04.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:04.809+0000 7fdb76733700 1 -- 172.21.15.88:0/715302863 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdb70060280 msgr2=0x7fdb70106760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:04.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:04.809+0000 7fdb76733700 1 --2- 172.21.15.88:0/715302863 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdb70060280 0x7fdb70106760 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7fdb600089f0 tx=0x7fdb6000b040).stop 2024-02-19T22:16:04.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:04.809+0000 7fdb76733700 1 -- 172.21.15.88:0/715302863 shutdown_connections 2024-02-19T22:16:04.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:04.809+0000 7fdb76733700 1 --2- 172.21.15.88:0/715302863 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdb5802ff10 0x7fdb580323d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:04.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:04.809+0000 7fdb76733700 1 --2- 172.21.15.88:0/715302863 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdb70060280 0x7fdb70106760 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:04.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:04.809+0000 7fdb76733700 1 -- 172.21.15.88:0/715302863 >> 172.21.15.88:0/715302863 conn(0x7fdb700f0b70 msgr2=0x7fdb700f15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:04.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:04.809+0000 7fdb76733700 1 -- 172.21.15.88:0/715302863 shutdown_connections 2024-02-19T22:16:04.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:04.809+0000 7fdb76733700 1 -- 172.21.15.88:0/715302863 wait complete. 2024-02-19T22:16:04.812 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:16:05.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:05 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:16:03.676561+0000 mgr.smithi052.ghqrtk (mgr.14162) 196 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:05.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:05 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:16:04.808455+0000 mon.smithi052 (mon.0) 231 : audit [DBG] from='client.? 172.21.15.88:0/715302863' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:06.424 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:06.424 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:07.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:07 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:16:05.676850+0000 mgr.smithi052.ghqrtk (mgr.14162) 197 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:08.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:08.319+0000 7f91dffff700 1 -- 172.21.15.88:0/119406280 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f91e00f51a0 msgr2=0x7f91e00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:08.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:08.319+0000 7f91dffff700 1 --2- 172.21.15.88:0/119406280 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f91e00f51a0 0x7f91e00f55c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f91d0009fb0 tx=0x7f91d0009b30).stop 2024-02-19T22:16:08.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:08.319+0000 7f91dffff700 1 -- 172.21.15.88:0/119406280 shutdown_connections 2024-02-19T22:16:08.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:08.319+0000 7f91dffff700 1 --2- 172.21.15.88:0/119406280 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f91e00f51a0 0x7f91e00f55c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:08.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:08.319+0000 7f91dffff700 1 -- 172.21.15.88:0/119406280 >> 172.21.15.88:0/119406280 conn(0x7f91e00f09a0 msgr2=0x7f91e00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:08.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:08.319+0000 7f91dffff700 1 -- 172.21.15.88:0/119406280 shutdown_connections 2024-02-19T22:16:08.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:08.320+0000 7f91dffff700 1 -- 172.21.15.88:0/119406280 wait complete. 2024-02-19T22:16:08.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:08.320+0000 7f91dffff700 1 Processor -- start 2024-02-19T22:16:08.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:08.320+0000 7f91dffff700 1 -- start start 2024-02-19T22:16:08.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:08.321+0000 7f91dffff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f91e00f51a0 0x7f91e01054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:08.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:08.321+0000 7f91dffff700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e01059c0 con 0x7f91e00f51a0 2024-02-19T22:16:08.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:08.321+0000 7f91deffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f91e00f51a0 0x7f91e01054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:08.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:08.321+0000 7f91deffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f91e00f51a0 0x7f91e01054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:43342/0 (socket says 172.21.15.88:43342) 2024-02-19T22:16:08.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:08.321+0000 7f91deffd700 1 -- 172.21.15.88:0/296091661 learned_addr learned my addr 172.21.15.88:0/296091661 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:16:08.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:08.321+0000 7f91deffd700 1 -- 172.21.15.88:0/296091661 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d0005040 con 0x7f91e00f51a0 2024-02-19T22:16:08.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:08.321+0000 7f91deffd700 1 --2- 172.21.15.88:0/296091661 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f91e00f51a0 0x7f91e01054d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f91d00089f0 tx=0x7f91d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:08.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:08.321+0000 7f91c77fe700 1 -- 172.21.15.88:0/296091661 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f91d0009320 con 0x7f91e00f51a0 2024-02-19T22:16:08.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:08.321+0000 7f91dffff700 1 -- 172.21.15.88:0/296091661 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91e0105bc0 con 0x7f91e00f51a0 2024-02-19T22:16:08.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:08.321+0000 7f91c77fe700 1 -- 172.21.15.88:0/296091661 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f91d0013380 con 0x7f91e00f51a0 2024-02-19T22:16:08.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:08.321+0000 7f91c77fe700 1 -- 172.21.15.88:0/296091661 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f91d0014e20 con 0x7f91e00f51a0 2024-02-19T22:16:08.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:08.321+0000 7f91dffff700 1 -- 172.21.15.88:0/296091661 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91e0106050 con 0x7f91e00f51a0 2024-02-19T22:16:08.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:08.322+0000 7f91c77fe700 1 -- 172.21.15.88:0/296091661 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f91d0014420 con 0x7f91e00f51a0 2024-02-19T22:16:08.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:08.322+0000 7f91dffff700 1 -- 172.21.15.88:0/296091661 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91e0045ad0 con 0x7f91e00f51a0 2024-02-19T22:16:08.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:08.322+0000 7f91c77fe700 1 --2- 172.21.15.88:0/296091661 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f91c802ff10 0x7f91c80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:08.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:08.322+0000 7f91c77fe700 1 -- 172.21.15.88:0/296091661 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f91d0040f90 con 0x7f91e00f51a0 2024-02-19T22:16:08.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:08.323+0000 7f91de7fc700 1 --2- 172.21.15.88:0/296091661 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f91c802ff10 0x7f91c80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:08.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:08.323+0000 7f91de7fc700 1 --2- 172.21.15.88:0/296091661 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f91c802ff10 0x7f91c80323d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f91d40096a0 tx=0x7f91d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:08.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:08.327+0000 7f91c77fe700 1 -- 172.21.15.88:0/296091661 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f91d0014790 con 0x7f91e00f51a0 2024-02-19T22:16:08.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:08.531+0000 7f91dffff700 1 -- 172.21.15.88:0/296091661 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f91e0106ea0 con 0x7f91e00f51a0 2024-02-19T22:16:08.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:08.532+0000 7f91c77fe700 1 -- 172.21.15.88:0/296091661 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f91d001d7f0 con 0x7f91e00f51a0 2024-02-19T22:16:08.534 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:16:08.534 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:08.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:08.533+0000 7f91dffff700 1 -- 172.21.15.88:0/296091661 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f91c802ff10 msgr2=0x7f91c80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:08.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:08.533+0000 7f91dffff700 1 --2- 172.21.15.88:0/296091661 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f91c802ff10 0x7f91c80323d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f91d40096a0 tx=0x7f91d4006b40).stop 2024-02-19T22:16:08.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:08.533+0000 7f91dffff700 1 -- 172.21.15.88:0/296091661 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f91e00f51a0 msgr2=0x7f91e01054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:08.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:08.533+0000 7f91dffff700 1 --2- 172.21.15.88:0/296091661 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f91e00f51a0 0x7f91e01054d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f91d00089f0 tx=0x7f91d000b040).stop 2024-02-19T22:16:08.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:08.534+0000 7f91dffff700 1 -- 172.21.15.88:0/296091661 shutdown_connections 2024-02-19T22:16:08.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:08.534+0000 7f91dffff700 1 --2- 172.21.15.88:0/296091661 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f91c802ff10 0x7f91c80323d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:08.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:08.534+0000 7f91dffff700 1 --2- 172.21.15.88:0/296091661 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f91e00f51a0 0x7f91e01054d0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:08.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:08.534+0000 7f91dffff700 1 -- 172.21.15.88:0/296091661 >> 172.21.15.88:0/296091661 conn(0x7f91e00f09a0 msgr2=0x7f91e00f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:08.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:08.534+0000 7f91dffff700 1 -- 172.21.15.88:0/296091661 shutdown_connections 2024-02-19T22:16:08.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:08.534+0000 7f91dffff700 1 -- 172.21.15.88:0/296091661 wait complete. 2024-02-19T22:16:08.537 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:16:09.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:09 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:16:07.677196+0000 mgr.smithi052.ghqrtk (mgr.14162) 198 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:09.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:09 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:16:08.533453+0000 mon.smithi052 (mon.0) 232 : audit [DBG] from='client.? 172.21.15.88:0/296091661' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:10.180 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:10.180 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:11.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:11 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:16:09.677565+0000 mgr.smithi052.ghqrtk (mgr.14162) 199 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:12.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:12.038+0000 7fec6946f700 1 -- 172.21.15.88:0/3152858182 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec640f51a0 msgr2=0x7fec640f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:12.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:12.038+0000 7fec6946f700 1 --2- 172.21.15.88:0/3152858182 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec640f51a0 0x7fec640f55c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7fec54009fb0 tx=0x7fec54009b30).stop 2024-02-19T22:16:12.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:12.038+0000 7fec6946f700 1 -- 172.21.15.88:0/3152858182 shutdown_connections 2024-02-19T22:16:12.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:12.038+0000 7fec6946f700 1 --2- 172.21.15.88:0/3152858182 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec640f51a0 0x7fec640f55c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:12.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:12.038+0000 7fec6946f700 1 -- 172.21.15.88:0/3152858182 >> 172.21.15.88:0/3152858182 conn(0x7fec640f09a0 msgr2=0x7fec640f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:12.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:12.038+0000 7fec6946f700 1 -- 172.21.15.88:0/3152858182 shutdown_connections 2024-02-19T22:16:12.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:12.038+0000 7fec6946f700 1 -- 172.21.15.88:0/3152858182 wait complete. 2024-02-19T22:16:12.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:12.038+0000 7fec6946f700 1 Processor -- start 2024-02-19T22:16:12.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:12.039+0000 7fec6946f700 1 -- start start 2024-02-19T22:16:12.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:12.039+0000 7fec6946f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec640f51a0 0x7fec641054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:12.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:12.039+0000 7fec6946f700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec641059c0 con 0x7fec640f51a0 2024-02-19T22:16:12.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:12.039+0000 7fec63fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec640f51a0 0x7fec641054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:12.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:12.039+0000 7fec63fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec640f51a0 0x7fec641054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:43366/0 (socket says 172.21.15.88:43366) 2024-02-19T22:16:12.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:12.039+0000 7fec63fff700 1 -- 172.21.15.88:0/1957914063 learned_addr learned my addr 172.21.15.88:0/1957914063 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:16:12.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:12.040+0000 7fec63fff700 1 -- 172.21.15.88:0/1957914063 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec54005040 con 0x7fec640f51a0 2024-02-19T22:16:12.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:12.040+0000 7fec63fff700 1 --2- 172.21.15.88:0/1957914063 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec640f51a0 0x7fec641054d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fec540089f0 tx=0x7fec5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:12.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:12.040+0000 7fec60ff9700 1 -- 172.21.15.88:0/1957914063 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec54009320 con 0x7fec640f51a0 2024-02-19T22:16:12.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:12.040+0000 7fec6946f700 1 -- 172.21.15.88:0/1957914063 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec64105bc0 con 0x7fec640f51a0 2024-02-19T22:16:12.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:12.040+0000 7fec60ff9700 1 -- 172.21.15.88:0/1957914063 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fec54013380 con 0x7fec640f51a0 2024-02-19T22:16:12.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:12.040+0000 7fec60ff9700 1 -- 172.21.15.88:0/1957914063 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec54014e20 con 0x7fec640f51a0 2024-02-19T22:16:12.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:12.040+0000 7fec6946f700 1 -- 172.21.15.88:0/1957914063 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec64106050 con 0x7fec640f51a0 2024-02-19T22:16:12.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:12.041+0000 7fec60ff9700 1 -- 172.21.15.88:0/1957914063 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fec540095f0 con 0x7fec640f51a0 2024-02-19T22:16:12.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:12.041+0000 7fec6946f700 1 -- 172.21.15.88:0/1957914063 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec64045ad0 con 0x7fec640f51a0 2024-02-19T22:16:12.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:12.041+0000 7fec60ff9700 1 --2- 172.21.15.88:0/1957914063 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fec4c02ff10 0x7fec4c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:12.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:12.041+0000 7fec60ff9700 1 -- 172.21.15.88:0/1957914063 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fec54040f40 con 0x7fec640f51a0 2024-02-19T22:16:12.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:12.041+0000 7fec637fe700 1 --2- 172.21.15.88:0/1957914063 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fec4c02ff10 0x7fec4c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:12.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:12.041+0000 7fec637fe700 1 --2- 172.21.15.88:0/1957914063 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fec4c02ff10 0x7fec4c0323d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fec580096a0 tx=0x7fec58006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:12.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:12.046+0000 7fec60ff9700 1 -- 172.21.15.88:0/1957914063 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fec54014420 con 0x7fec640f51a0 2024-02-19T22:16:12.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:12.250+0000 7fec6946f700 1 -- 172.21.15.88:0/1957914063 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fec64106f00 con 0x7fec640f51a0 2024-02-19T22:16:12.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:12.251+0000 7fec60ff9700 1 -- 172.21.15.88:0/1957914063 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fec54010230 con 0x7fec640f51a0 2024-02-19T22:16:12.253 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:16:12.253 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:12.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:12.252+0000 7fec6946f700 1 -- 172.21.15.88:0/1957914063 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fec4c02ff10 msgr2=0x7fec4c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:12.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:12.252+0000 7fec6946f700 1 --2- 172.21.15.88:0/1957914063 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fec4c02ff10 0x7fec4c0323d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fec580096a0 tx=0x7fec58006b40).stop 2024-02-19T22:16:12.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:12.252+0000 7fec6946f700 1 -- 172.21.15.88:0/1957914063 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec640f51a0 msgr2=0x7fec641054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:12.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:12.252+0000 7fec6946f700 1 --2- 172.21.15.88:0/1957914063 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec640f51a0 0x7fec641054d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fec540089f0 tx=0x7fec5400b040).stop 2024-02-19T22:16:12.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:12.252+0000 7fec6946f700 1 -- 172.21.15.88:0/1957914063 shutdown_connections 2024-02-19T22:16:12.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:12.252+0000 7fec6946f700 1 --2- 172.21.15.88:0/1957914063 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fec4c02ff10 0x7fec4c0323d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:12.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:12.252+0000 7fec6946f700 1 --2- 172.21.15.88:0/1957914063 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec640f51a0 0x7fec641054d0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:12.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:12.252+0000 7fec6946f700 1 -- 172.21.15.88:0/1957914063 >> 172.21.15.88:0/1957914063 conn(0x7fec640f09a0 msgr2=0x7fec640f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:12.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:12.253+0000 7fec6946f700 1 -- 172.21.15.88:0/1957914063 shutdown_connections 2024-02-19T22:16:12.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:12.253+0000 7fec6946f700 1 -- 172.21.15.88:0/1957914063 wait complete. 2024-02-19T22:16:12.256 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:16:12.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:12 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:16:12.252318+0000 mon.smithi052 (mon.0) 233 : audit [DBG] from='client.? 172.21.15.88:0/1957914063' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:13.833 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:13.833 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:13.842 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:13 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:16:11.677823+0000 mgr.smithi052.ghqrtk (mgr.14162) 200 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:15.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:15.574+0000 7fec25647700 1 -- 172.21.15.88:0/3288920771 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec200f6510 msgr2=0x7fec200f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:15.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:15.574+0000 7fec25647700 1 --2- 172.21.15.88:0/3288920771 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec200f6510 0x7fec200f6930 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7fec14009fb0 tx=0x7fec14009b30).stop 2024-02-19T22:16:15.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:15.574+0000 7fec25647700 1 -- 172.21.15.88:0/3288920771 shutdown_connections 2024-02-19T22:16:15.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:15.574+0000 7fec25647700 1 --2- 172.21.15.88:0/3288920771 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec200f6510 0x7fec200f6930 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:15.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:15.574+0000 7fec25647700 1 -- 172.21.15.88:0/3288920771 >> 172.21.15.88:0/3288920771 conn(0x7fec200f1c70 msgr2=0x7fec200f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:15.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:15.574+0000 7fec25647700 1 -- 172.21.15.88:0/3288920771 shutdown_connections 2024-02-19T22:16:15.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:15.574+0000 7fec25647700 1 -- 172.21.15.88:0/3288920771 wait complete. 2024-02-19T22:16:15.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:15.575+0000 7fec25647700 1 Processor -- start 2024-02-19T22:16:15.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:15.575+0000 7fec25647700 1 -- start start 2024-02-19T22:16:15.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:15.575+0000 7fec25647700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec200f6510 0x7fec201089c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:15.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:15.575+0000 7fec25647700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec20108eb0 con 0x7fec200f6510 2024-02-19T22:16:15.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:15.576+0000 7fec1effd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec200f6510 0x7fec201089c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:15.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:15.576+0000 7fec1effd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec200f6510 0x7fec201089c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:50018/0 (socket says 172.21.15.88:50018) 2024-02-19T22:16:15.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:15.576+0000 7fec1effd700 1 -- 172.21.15.88:0/1671017651 learned_addr learned my addr 172.21.15.88:0/1671017651 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:16:15.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:15.576+0000 7fec1effd700 1 -- 172.21.15.88:0/1671017651 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec14005040 con 0x7fec200f6510 2024-02-19T22:16:15.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:15.576+0000 7fec1effd700 1 --2- 172.21.15.88:0/1671017651 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec200f6510 0x7fec201089c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7fec14005010 tx=0x7fec1400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:15.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:15.576+0000 7fec077fe700 1 -- 172.21.15.88:0/1671017651 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec14009320 con 0x7fec200f6510 2024-02-19T22:16:15.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:15.576+0000 7fec25647700 1 -- 172.21.15.88:0/1671017651 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec201090b0 con 0x7fec200f6510 2024-02-19T22:16:15.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:15.576+0000 7fec077fe700 1 -- 172.21.15.88:0/1671017651 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fec14013380 con 0x7fec200f6510 2024-02-19T22:16:15.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:15.576+0000 7fec077fe700 1 -- 172.21.15.88:0/1671017651 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec14014e50 con 0x7fec200f6510 2024-02-19T22:16:15.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:15.576+0000 7fec25647700 1 -- 172.21.15.88:0/1671017651 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec20109540 con 0x7fec200f6510 2024-02-19T22:16:15.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:15.577+0000 7fec077fe700 1 -- 172.21.15.88:0/1671017651 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fec140263a0 con 0x7fec200f6510 2024-02-19T22:16:15.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:15.577+0000 7fec25647700 1 -- 172.21.15.88:0/1671017651 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec20045ad0 con 0x7fec200f6510 2024-02-19T22:16:15.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:15.577+0000 7fec077fe700 1 --2- 172.21.15.88:0/1671017651 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fec0802ff10 0x7fec080323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:15.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:15.577+0000 7fec077fe700 1 -- 172.21.15.88:0/1671017651 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fec14041010 con 0x7fec200f6510 2024-02-19T22:16:15.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:15.577+0000 7fec1e7fc700 1 --2- 172.21.15.88:0/1671017651 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fec0802ff10 0x7fec080323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:15.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:15.578+0000 7fec1e7fc700 1 --2- 172.21.15.88:0/1671017651 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fec0802ff10 0x7fec080323d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fec100096a0 tx=0x7fec10006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:15.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:15.582+0000 7fec077fe700 1 -- 172.21.15.88:0/1671017651 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fec1401dde0 con 0x7fec200f6510 2024-02-19T22:16:15.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:15.787+0000 7fec25647700 1 -- 172.21.15.88:0/1671017651 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fec2006b150 con 0x7fec200f6510 2024-02-19T22:16:15.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:15.787+0000 7fec077fe700 1 -- 172.21.15.88:0/1671017651 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fec1401d9d0 con 0x7fec200f6510 2024-02-19T22:16:15.788 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:16:15.789 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:15.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:15.788+0000 7fec25647700 1 -- 172.21.15.88:0/1671017651 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fec0802ff10 msgr2=0x7fec080323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:15.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:15.788+0000 7fec25647700 1 --2- 172.21.15.88:0/1671017651 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fec0802ff10 0x7fec080323d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fec100096a0 tx=0x7fec10006b40).stop 2024-02-19T22:16:15.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:15.788+0000 7fec25647700 1 -- 172.21.15.88:0/1671017651 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec200f6510 msgr2=0x7fec201089c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:15.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:15.788+0000 7fec25647700 1 --2- 172.21.15.88:0/1671017651 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec200f6510 0x7fec201089c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7fec14005010 tx=0x7fec1400b040).stop 2024-02-19T22:16:15.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:15.788+0000 7fec25647700 1 -- 172.21.15.88:0/1671017651 shutdown_connections 2024-02-19T22:16:15.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:15.788+0000 7fec25647700 1 --2- 172.21.15.88:0/1671017651 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fec0802ff10 0x7fec080323d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:15.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:15.789+0000 7fec25647700 1 --2- 172.21.15.88:0/1671017651 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec200f6510 0x7fec201089c0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:15.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:15.789+0000 7fec25647700 1 -- 172.21.15.88:0/1671017651 >> 172.21.15.88:0/1671017651 conn(0x7fec200f1c70 msgr2=0x7fec200f29c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:15.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:15.789+0000 7fec25647700 1 -- 172.21.15.88:0/1671017651 shutdown_connections 2024-02-19T22:16:15.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:15.789+0000 7fec25647700 1 -- 172.21.15.88:0/1671017651 wait complete. 2024-02-19T22:16:15.791 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:16:15.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:15 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:16:13.678185+0000 mgr.smithi052.ghqrtk (mgr.14162) 201 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:16.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:16 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:16:15.787558+0000 mon.smithi052 (mon.0) 234 : audit [DBG] from='client.? 172.21.15.88:0/1671017651' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:17.397 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:17.397 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:17.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:17 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:16:15.678553+0000 mgr.smithi052.ghqrtk (mgr.14162) 202 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:19.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:19.313+0000 7fe0d3644700 1 -- 172.21.15.88:0/134903399 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe0cc0f5410 msgr2=0x7fe0cc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:19.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:19.313+0000 7fe0d3644700 1 --2- 172.21.15.88:0/134903399 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe0cc0f5410 0x7fe0cc0f5830 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7fe0bc009fb0 tx=0x7fe0bc009b30).stop 2024-02-19T22:16:19.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:19.313+0000 7fe0d3644700 1 -- 172.21.15.88:0/134903399 shutdown_connections 2024-02-19T22:16:19.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:19.313+0000 7fe0d3644700 1 --2- 172.21.15.88:0/134903399 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe0cc0f5410 0x7fe0cc0f5830 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:19.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:19.313+0000 7fe0d3644700 1 -- 172.21.15.88:0/134903399 >> 172.21.15.88:0/134903399 conn(0x7fe0cc0f0b70 msgr2=0x7fe0cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:19.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:19.313+0000 7fe0d3644700 1 -- 172.21.15.88:0/134903399 shutdown_connections 2024-02-19T22:16:19.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:19.314+0000 7fe0d3644700 1 -- 172.21.15.88:0/134903399 wait complete. 2024-02-19T22:16:19.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:19.314+0000 7fe0d3644700 1 Processor -- start 2024-02-19T22:16:19.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:19.314+0000 7fe0d3644700 1 -- start start 2024-02-19T22:16:19.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:19.314+0000 7fe0d3644700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe0cc0f5410 0x7fe0cc1094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:19.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:19.314+0000 7fe0d3644700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0cc060240 con 0x7fe0cc0f5410 2024-02-19T22:16:19.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:19.314+0000 7fe0d13e0700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe0cc0f5410 0x7fe0cc1094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:19.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:19.314+0000 7fe0d13e0700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe0cc0f5410 0x7fe0cc1094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:50036/0 (socket says 172.21.15.88:50036) 2024-02-19T22:16:19.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:19.314+0000 7fe0d13e0700 1 -- 172.21.15.88:0/245873819 learned_addr learned my addr 172.21.15.88:0/245873819 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:16:19.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:19.315+0000 7fe0d13e0700 1 -- 172.21.15.88:0/245873819 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0bc005040 con 0x7fe0cc0f5410 2024-02-19T22:16:19.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:19.315+0000 7fe0d13e0700 1 --2- 172.21.15.88:0/245873819 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe0cc0f5410 0x7fe0cc1094e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7fe0bc005010 tx=0x7fe0bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:19.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:19.315+0000 7fe0c1ffb700 1 -- 172.21.15.88:0/245873819 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0bc009320 con 0x7fe0cc0f5410 2024-02-19T22:16:19.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:19.315+0000 7fe0c1ffb700 1 -- 172.21.15.88:0/245873819 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe0bc013380 con 0x7fe0cc0f5410 2024-02-19T22:16:19.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:19.315+0000 7fe0d3644700 1 -- 172.21.15.88:0/245873819 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0cc060440 con 0x7fe0cc0f5410 2024-02-19T22:16:19.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:19.315+0000 7fe0c1ffb700 1 -- 172.21.15.88:0/245873819 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0bc014e50 con 0x7fe0cc0f5410 2024-02-19T22:16:19.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:19.315+0000 7fe0d3644700 1 -- 172.21.15.88:0/245873819 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0cc109b90 con 0x7fe0cc0f5410 2024-02-19T22:16:19.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:19.316+0000 7fe0d3644700 1 -- 172.21.15.88:0/245873819 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0cc045ad0 con 0x7fe0cc0f5410 2024-02-19T22:16:19.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:19.316+0000 7fe0c1ffb700 1 -- 172.21.15.88:0/245873819 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe0bc0095f0 con 0x7fe0cc0f5410 2024-02-19T22:16:19.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:19.316+0000 7fe0c1ffb700 1 --2- 172.21.15.88:0/245873819 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fe0b802ff10 0x7fe0b80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:19.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:19.316+0000 7fe0c1ffb700 1 -- 172.21.15.88:0/245873819 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe0bc040e70 con 0x7fe0cc0f5410 2024-02-19T22:16:19.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:19.317+0000 7fe0d0bdf700 1 --2- 172.21.15.88:0/245873819 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fe0b802ff10 0x7fe0b80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:19.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:19.317+0000 7fe0d0bdf700 1 --2- 172.21.15.88:0/245873819 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fe0b802ff10 0x7fe0b80323d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fe0c80096a0 tx=0x7fe0c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:19.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:19.321+0000 7fe0c1ffb700 1 -- 172.21.15.88:0/245873819 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe0bc014420 con 0x7fe0cc0f5410 2024-02-19T22:16:19.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:19.524+0000 7fe0d3644700 1 -- 172.21.15.88:0/245873819 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe0cc10a3a0 con 0x7fe0cc0f5410 2024-02-19T22:16:19.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:19.524+0000 7fe0c1ffb700 1 -- 172.21.15.88:0/245873819 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe0bc010230 con 0x7fe0cc0f5410 2024-02-19T22:16:19.526 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:16:19.526 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:19.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:19.526+0000 7fe0d3644700 1 -- 172.21.15.88:0/245873819 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fe0b802ff10 msgr2=0x7fe0b80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:19.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:19.526+0000 7fe0d3644700 1 --2- 172.21.15.88:0/245873819 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fe0b802ff10 0x7fe0b80323d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fe0c80096a0 tx=0x7fe0c8006b40).stop 2024-02-19T22:16:19.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:19.526+0000 7fe0d3644700 1 -- 172.21.15.88:0/245873819 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe0cc0f5410 msgr2=0x7fe0cc1094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:19.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:19.526+0000 7fe0d3644700 1 --2- 172.21.15.88:0/245873819 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe0cc0f5410 0x7fe0cc1094e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7fe0bc005010 tx=0x7fe0bc00b040).stop 2024-02-19T22:16:19.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:19.526+0000 7fe0d3644700 1 -- 172.21.15.88:0/245873819 shutdown_connections 2024-02-19T22:16:19.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:19.526+0000 7fe0d3644700 1 --2- 172.21.15.88:0/245873819 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fe0b802ff10 0x7fe0b80323d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:19.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:19.526+0000 7fe0d3644700 1 --2- 172.21.15.88:0/245873819 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe0cc0f5410 0x7fe0cc1094e0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:19.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:19.526+0000 7fe0d3644700 1 -- 172.21.15.88:0/245873819 >> 172.21.15.88:0/245873819 conn(0x7fe0cc0f0b70 msgr2=0x7fe0cc0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:19.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:19.526+0000 7fe0d3644700 1 -- 172.21.15.88:0/245873819 shutdown_connections 2024-02-19T22:16:19.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:19.526+0000 7fe0d3644700 1 -- 172.21.15.88:0/245873819 wait complete. 2024-02-19T22:16:19.528 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:16:19.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:19 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:16:17.678853+0000 mgr.smithi052.ghqrtk (mgr.14162) 203 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:20.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:20 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:16:19.524976+0000 mon.smithi052 (mon.0) 235 : audit [DBG] from='client.? 172.21.15.88:0/245873819' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:21.149 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:21.150 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:21.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:21 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:16:19.679188+0000 mgr.smithi052.ghqrtk (mgr.14162) 204 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:23.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:23.098+0000 7f772539c700 1 -- 172.21.15.88:0/411203244 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f77200f6510 msgr2=0x7f77200f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:23.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:23.098+0000 7f772539c700 1 --2- 172.21.15.88:0/411203244 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f77200f6510 0x7f77200f6930 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f7710009fb0 tx=0x7f7710009b30).stop 2024-02-19T22:16:23.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:23.098+0000 7f772539c700 1 -- 172.21.15.88:0/411203244 shutdown_connections 2024-02-19T22:16:23.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:23.098+0000 7f772539c700 1 --2- 172.21.15.88:0/411203244 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f77200f6510 0x7f77200f6930 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:23.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:23.098+0000 7f772539c700 1 -- 172.21.15.88:0/411203244 >> 172.21.15.88:0/411203244 conn(0x7f77200f1c70 msgr2=0x7f77200f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:23.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:23.098+0000 7f772539c700 1 -- 172.21.15.88:0/411203244 shutdown_connections 2024-02-19T22:16:23.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:23.098+0000 7f772539c700 1 -- 172.21.15.88:0/411203244 wait complete. 2024-02-19T22:16:23.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:23.099+0000 7f772539c700 1 Processor -- start 2024-02-19T22:16:23.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:23.099+0000 7f772539c700 1 -- start start 2024-02-19T22:16:23.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:23.100+0000 7f772539c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f77200f6510 0x7f77201067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:23.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:23.100+0000 7f772539c700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7720106cd0 con 0x7f77200f6510 2024-02-19T22:16:23.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:23.100+0000 7f771effd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f77200f6510 0x7f77201067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:23.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:23.100+0000 7f771effd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f77200f6510 0x7f77201067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:50056/0 (socket says 172.21.15.88:50056) 2024-02-19T22:16:23.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:23.100+0000 7f771effd700 1 -- 172.21.15.88:0/1867913425 learned_addr learned my addr 172.21.15.88:0/1867913425 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:16:23.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:23.100+0000 7f771effd700 1 -- 172.21.15.88:0/1867913425 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7710005040 con 0x7f77200f6510 2024-02-19T22:16:23.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:23.100+0000 7f771effd700 1 --2- 172.21.15.88:0/1867913425 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f77200f6510 0x7f77201067e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f7710005010 tx=0x7f771000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:23.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:23.100+0000 7f77077fe700 1 -- 172.21.15.88:0/1867913425 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7710009320 con 0x7f77200f6510 2024-02-19T22:16:23.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:23.100+0000 7f772539c700 1 -- 172.21.15.88:0/1867913425 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7720106ed0 con 0x7f77200f6510 2024-02-19T22:16:23.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:23.100+0000 7f77077fe700 1 -- 172.21.15.88:0/1867913425 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7710013380 con 0x7f77200f6510 2024-02-19T22:16:23.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:23.100+0000 7f77077fe700 1 -- 172.21.15.88:0/1867913425 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7710014e50 con 0x7f77200f6510 2024-02-19T22:16:23.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:23.100+0000 7f772539c700 1 -- 172.21.15.88:0/1867913425 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7720107360 con 0x7f77200f6510 2024-02-19T22:16:23.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:23.101+0000 7f77077fe700 1 -- 172.21.15.88:0/1867913425 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f77100095f0 con 0x7f77200f6510 2024-02-19T22:16:23.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:23.101+0000 7f772539c700 1 -- 172.21.15.88:0/1867913425 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7720045ad0 con 0x7f77200f6510 2024-02-19T22:16:23.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:23.101+0000 7f77077fe700 1 --2- 172.21.15.88:0/1867913425 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f770802ff10 0x7f77080323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:23.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:23.101+0000 7f77077fe700 1 -- 172.21.15.88:0/1867913425 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7710040e90 con 0x7f77200f6510 2024-02-19T22:16:23.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:23.102+0000 7f771e7fc700 1 --2- 172.21.15.88:0/1867913425 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f770802ff10 0x7f77080323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:23.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:23.102+0000 7f771e7fc700 1 --2- 172.21.15.88:0/1867913425 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f770802ff10 0x7f77080323d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f77140096a0 tx=0x7f7714006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:23.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:23.106+0000 7f77077fe700 1 -- 172.21.15.88:0/1867913425 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7710014420 con 0x7f77200f6510 2024-02-19T22:16:23.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:23.311+0000 7f772539c700 1 -- 172.21.15.88:0/1867913425 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7720108210 con 0x7f77200f6510 2024-02-19T22:16:23.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:23.312+0000 7f77077fe700 1 -- 172.21.15.88:0/1867913425 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7710010230 con 0x7f77200f6510 2024-02-19T22:16:23.313 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:16:23.313 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:23.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:23.314+0000 7f772539c700 1 -- 172.21.15.88:0/1867913425 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f770802ff10 msgr2=0x7f77080323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:23.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:23.314+0000 7f772539c700 1 --2- 172.21.15.88:0/1867913425 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f770802ff10 0x7f77080323d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f77140096a0 tx=0x7f7714006b40).stop 2024-02-19T22:16:23.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:23.314+0000 7f772539c700 1 -- 172.21.15.88:0/1867913425 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f77200f6510 msgr2=0x7f77201067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:23.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:23.314+0000 7f772539c700 1 --2- 172.21.15.88:0/1867913425 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f77200f6510 0x7f77201067e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f7710005010 tx=0x7f771000b040).stop 2024-02-19T22:16:23.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:23.314+0000 7f772539c700 1 -- 172.21.15.88:0/1867913425 shutdown_connections 2024-02-19T22:16:23.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:23.314+0000 7f772539c700 1 --2- 172.21.15.88:0/1867913425 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f770802ff10 0x7f77080323d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:23.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:23.314+0000 7f772539c700 1 --2- 172.21.15.88:0/1867913425 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f77200f6510 0x7f77201067e0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:23.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:23.314+0000 7f772539c700 1 -- 172.21.15.88:0/1867913425 >> 172.21.15.88:0/1867913425 conn(0x7f77200f1c70 msgr2=0x7f77200f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:23.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:23.314+0000 7f772539c700 1 -- 172.21.15.88:0/1867913425 shutdown_connections 2024-02-19T22:16:23.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:23.314+0000 7f772539c700 1 -- 172.21.15.88:0/1867913425 wait complete. 2024-02-19T22:16:23.316 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:16:23.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:23 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:16:21.679447+0000 mgr.smithi052.ghqrtk (mgr.14162) 205 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:23.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:23 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:16:23.312584+0000 mon.smithi052 (mon.0) 236 : audit [DBG] from='client.? 172.21.15.88:0/1867913425' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:24.846 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:24.846 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:25.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:25 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:16:23.679773+0000 mgr.smithi052.ghqrtk (mgr.14162) 206 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:26.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:26.894+0000 7f16469a2700 1 -- 172.21.15.88:0/2588499089 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f16400f51a0 msgr2=0x7f16400f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:26.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:26.894+0000 7f16469a2700 1 --2- 172.21.15.88:0/2588499089 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f16400f51a0 0x7f16400f55c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f1630009fb0 tx=0x7f1630009b30).stop 2024-02-19T22:16:26.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:26.894+0000 7f16469a2700 1 -- 172.21.15.88:0/2588499089 shutdown_connections 2024-02-19T22:16:26.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:26.894+0000 7f16469a2700 1 --2- 172.21.15.88:0/2588499089 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f16400f51a0 0x7f16400f55c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:26.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:26.894+0000 7f16469a2700 1 -- 172.21.15.88:0/2588499089 >> 172.21.15.88:0/2588499089 conn(0x7f16400f09a0 msgr2=0x7f16400f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:26.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:26.894+0000 7f16469a2700 1 -- 172.21.15.88:0/2588499089 shutdown_connections 2024-02-19T22:16:26.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:26.894+0000 7f16469a2700 1 -- 172.21.15.88:0/2588499089 wait complete. 2024-02-19T22:16:26.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:26.895+0000 7f16469a2700 1 Processor -- start 2024-02-19T22:16:26.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:26.895+0000 7f16469a2700 1 -- start start 2024-02-19T22:16:26.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:26.895+0000 7f16469a2700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f16400f51a0 0x7f16401065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:26.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:26.895+0000 7f16469a2700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1640106ac0 con 0x7f16400f51a0 2024-02-19T22:16:26.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:26.895+0000 7f16459a0700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f16400f51a0 0x7f16401065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:26.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:26.895+0000 7f16459a0700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f16400f51a0 0x7f16401065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:48504/0 (socket says 172.21.15.88:48504) 2024-02-19T22:16:26.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:26.895+0000 7f16459a0700 1 -- 172.21.15.88:0/119635967 learned_addr learned my addr 172.21.15.88:0/119635967 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:16:26.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:26.896+0000 7f16459a0700 1 -- 172.21.15.88:0/119635967 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1630005040 con 0x7f16400f51a0 2024-02-19T22:16:26.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:26.896+0000 7f16459a0700 1 --2- 172.21.15.88:0/119635967 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f16400f51a0 0x7f16401065d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f16300089f0 tx=0x7f163000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:26.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:26.896+0000 7f16367fc700 1 -- 172.21.15.88:0/119635967 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1630009320 con 0x7f16400f51a0 2024-02-19T22:16:26.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:26.896+0000 7f16367fc700 1 -- 172.21.15.88:0/119635967 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1630013380 con 0x7f16400f51a0 2024-02-19T22:16:26.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:26.896+0000 7f16469a2700 1 -- 172.21.15.88:0/119635967 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1640106cc0 con 0x7f16400f51a0 2024-02-19T22:16:26.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:26.896+0000 7f16367fc700 1 -- 172.21.15.88:0/119635967 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1630014e20 con 0x7f16400f51a0 2024-02-19T22:16:26.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:26.896+0000 7f16469a2700 1 -- 172.21.15.88:0/119635967 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1640107150 con 0x7f16400f51a0 2024-02-19T22:16:26.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:26.897+0000 7f16367fc700 1 -- 172.21.15.88:0/119635967 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f16300263a0 con 0x7f16400f51a0 2024-02-19T22:16:26.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:26.897+0000 7f16367fc700 1 --2- 172.21.15.88:0/119635967 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f162c02ff10 0x7f162c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:26.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:26.897+0000 7f16469a2700 1 -- 172.21.15.88:0/119635967 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1640045ad0 con 0x7f16400f51a0 2024-02-19T22:16:26.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:26.897+0000 7f16367fc700 1 -- 172.21.15.88:0/119635967 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1630040ee0 con 0x7f16400f51a0 2024-02-19T22:16:26.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:26.897+0000 7f164519f700 1 --2- 172.21.15.88:0/119635967 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f162c02ff10 0x7f162c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:26.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:26.897+0000 7f164519f700 1 --2- 172.21.15.88:0/119635967 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f162c02ff10 0x7f162c0323d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f163c0096a0 tx=0x7f163c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:26.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:26.902+0000 7f16367fc700 1 -- 172.21.15.88:0/119635967 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1630026630 con 0x7f16400f51a0 2024-02-19T22:16:27.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:27.108+0000 7f16469a2700 1 -- 172.21.15.88:0/119635967 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f16400fad30 con 0x7f16400f51a0 2024-02-19T22:16:27.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:27.109+0000 7f16367fc700 1 -- 172.21.15.88:0/119635967 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f163001db70 con 0x7f16400f51a0 2024-02-19T22:16:27.111 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:16:27.112 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:27.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:27.110+0000 7f16469a2700 1 -- 172.21.15.88:0/119635967 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f162c02ff10 msgr2=0x7f162c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:27.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:27.110+0000 7f16469a2700 1 --2- 172.21.15.88:0/119635967 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f162c02ff10 0x7f162c0323d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f163c0096a0 tx=0x7f163c006b40).stop 2024-02-19T22:16:27.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:27.110+0000 7f16469a2700 1 -- 172.21.15.88:0/119635967 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f16400f51a0 msgr2=0x7f16401065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:27.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:27.110+0000 7f16469a2700 1 --2- 172.21.15.88:0/119635967 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f16400f51a0 0x7f16401065d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f16300089f0 tx=0x7f163000b040).stop 2024-02-19T22:16:27.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:27.110+0000 7f16469a2700 1 -- 172.21.15.88:0/119635967 shutdown_connections 2024-02-19T22:16:27.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:27.110+0000 7f16469a2700 1 --2- 172.21.15.88:0/119635967 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f162c02ff10 0x7f162c0323d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:27.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:27.110+0000 7f16469a2700 1 --2- 172.21.15.88:0/119635967 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f16400f51a0 0x7f16401065d0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:27.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:27.110+0000 7f16469a2700 1 -- 172.21.15.88:0/119635967 >> 172.21.15.88:0/119635967 conn(0x7f16400f09a0 msgr2=0x7f164005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:27.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:27.110+0000 7f16469a2700 1 -- 172.21.15.88:0/119635967 shutdown_connections 2024-02-19T22:16:27.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:27.110+0000 7f16469a2700 1 -- 172.21.15.88:0/119635967 wait complete. 2024-02-19T22:16:27.113 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:16:27.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:27 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:16:25.680162+0000 mgr.smithi052.ghqrtk (mgr.14162) 207 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:27.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:27 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:16:27.110347+0000 mon.smithi052 (mon.0) 237 : audit [DBG] from='client.? 172.21.15.88:0/119635967' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:28.779 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:28.780 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:29.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:29 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:16:27.680561+0000 mgr.smithi052.ghqrtk (mgr.14162) 208 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:30.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:30.779+0000 7fad1c817700 1 -- 172.21.15.88:0/1157928414 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fad180f5410 msgr2=0x7fad180f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:30.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:30.779+0000 7fad1c817700 1 --2- 172.21.15.88:0/1157928414 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fad180f5410 0x7fad180f5830 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7fad00009fb0 tx=0x7fad00009b30).stop 2024-02-19T22:16:30.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:30.779+0000 7fad1c817700 1 -- 172.21.15.88:0/1157928414 shutdown_connections 2024-02-19T22:16:30.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:30.779+0000 7fad1c817700 1 --2- 172.21.15.88:0/1157928414 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fad180f5410 0x7fad180f5830 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:30.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:30.779+0000 7fad1c817700 1 -- 172.21.15.88:0/1157928414 >> 172.21.15.88:0/1157928414 conn(0x7fad180f0b70 msgr2=0x7fad180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:30.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:30.779+0000 7fad1c817700 1 -- 172.21.15.88:0/1157928414 shutdown_connections 2024-02-19T22:16:30.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:30.779+0000 7fad1c817700 1 -- 172.21.15.88:0/1157928414 wait complete. 2024-02-19T22:16:30.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:30.780+0000 7fad1c817700 1 Processor -- start 2024-02-19T22:16:30.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:30.780+0000 7fad1c817700 1 -- start start 2024-02-19T22:16:30.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:30.780+0000 7fad1c817700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fad180f5410 0x7fad181056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:30.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:30.780+0000 7fad1c817700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad18105bd0 con 0x7fad180f5410 2024-02-19T22:16:30.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:30.780+0000 7fad1659c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fad180f5410 0x7fad181056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:30.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:30.780+0000 7fad1659c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fad180f5410 0x7fad181056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:48528/0 (socket says 172.21.15.88:48528) 2024-02-19T22:16:30.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:30.780+0000 7fad1659c700 1 -- 172.21.15.88:0/2303042770 learned_addr learned my addr 172.21.15.88:0/2303042770 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:16:30.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:30.781+0000 7fad1659c700 1 -- 172.21.15.88:0/2303042770 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad00005040 con 0x7fad180f5410 2024-02-19T22:16:30.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:30.781+0000 7fad1659c700 1 --2- 172.21.15.88:0/2303042770 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fad180f5410 0x7fad181056e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7fad00005010 tx=0x7fad0000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:30.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:30.781+0000 7fad0effd700 1 -- 172.21.15.88:0/2303042770 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad00009320 con 0x7fad180f5410 2024-02-19T22:16:30.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:30.781+0000 7fad0effd700 1 -- 172.21.15.88:0/2303042770 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fad00013380 con 0x7fad180f5410 2024-02-19T22:16:30.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:30.781+0000 7fad1c817700 1 -- 172.21.15.88:0/2303042770 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad18105dd0 con 0x7fad180f5410 2024-02-19T22:16:30.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:30.781+0000 7fad0effd700 1 -- 172.21.15.88:0/2303042770 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad00014e50 con 0x7fad180f5410 2024-02-19T22:16:30.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:30.781+0000 7fad1c817700 1 -- 172.21.15.88:0/2303042770 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad18106260 con 0x7fad180f5410 2024-02-19T22:16:30.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:30.782+0000 7fad0effd700 1 -- 172.21.15.88:0/2303042770 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fad000263a0 con 0x7fad180f5410 2024-02-19T22:16:30.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:30.782+0000 7fad1c817700 1 -- 172.21.15.88:0/2303042770 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad18045ad0 con 0x7fad180f5410 2024-02-19T22:16:30.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:30.782+0000 7fad0effd700 1 --2- 172.21.15.88:0/2303042770 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fad0402ff10 0x7fad040323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:30.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:30.782+0000 7fad0effd700 1 -- 172.21.15.88:0/2303042770 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fad000411a0 con 0x7fad180f5410 2024-02-19T22:16:30.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:30.782+0000 7fad15d9b700 1 --2- 172.21.15.88:0/2303042770 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fad0402ff10 0x7fad040323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:30.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:30.783+0000 7fad15d9b700 1 --2- 172.21.15.88:0/2303042770 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fad0402ff10 0x7fad040323d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fad080096a0 tx=0x7fad08006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:30.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:30.787+0000 7fad0effd700 1 -- 172.21.15.88:0/2303042770 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fad0001cb10 con 0x7fad180f5410 2024-02-19T22:16:30.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:30.996+0000 7fad1c817700 1 -- 172.21.15.88:0/2303042770 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fad18107110 con 0x7fad180f5410 2024-02-19T22:16:30.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:30.997+0000 7fad0effd700 1 -- 172.21.15.88:0/2303042770 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fad0001cb10 con 0x7fad180f5410 2024-02-19T22:16:30.999 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:16:30.999 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:31.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:30.999+0000 7fad1c817700 1 -- 172.21.15.88:0/2303042770 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fad0402ff10 msgr2=0x7fad040323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:31.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:30.999+0000 7fad1c817700 1 --2- 172.21.15.88:0/2303042770 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fad0402ff10 0x7fad040323d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fad080096a0 tx=0x7fad08006b40).stop 2024-02-19T22:16:31.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:30.999+0000 7fad1c817700 1 -- 172.21.15.88:0/2303042770 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fad180f5410 msgr2=0x7fad181056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:31.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:30.999+0000 7fad1c817700 1 --2- 172.21.15.88:0/2303042770 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fad180f5410 0x7fad181056e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7fad00005010 tx=0x7fad0000b040).stop 2024-02-19T22:16:31.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:30.999+0000 7fad1c817700 1 -- 172.21.15.88:0/2303042770 shutdown_connections 2024-02-19T22:16:31.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:30.999+0000 7fad1c817700 1 --2- 172.21.15.88:0/2303042770 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fad0402ff10 0x7fad040323d0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:31.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:30.999+0000 7fad1c817700 1 --2- 172.21.15.88:0/2303042770 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fad180f5410 0x7fad181056e0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:31.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:30.999+0000 7fad1c817700 1 -- 172.21.15.88:0/2303042770 >> 172.21.15.88:0/2303042770 conn(0x7fad180f0b70 msgr2=0x7fad180f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:31.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:30.999+0000 7fad1c817700 1 -- 172.21.15.88:0/2303042770 shutdown_connections 2024-02-19T22:16:31.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:30.999+0000 7fad1c817700 1 -- 172.21.15.88:0/2303042770 wait complete. 2024-02-19T22:16:31.002 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:16:31.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:31 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:16:29.680920+0000 mgr.smithi052.ghqrtk (mgr.14162) 209 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:31.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:31 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:16:30.998548+0000 mon.smithi052 (mon.0) 238 : audit [DBG] from='client.? 172.21.15.88:0/2303042770' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:32.743 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:32.744 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:33.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:33 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:16:31.681297+0000 mgr.smithi052.ghqrtk (mgr.14162) 210 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:34.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:34.689+0000 7ff282995700 1 -- 172.21.15.88:0/1230779265 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff27c0f51a0 msgr2=0x7ff27c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:34.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:34.689+0000 7ff282995700 1 --2- 172.21.15.88:0/1230779265 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff27c0f51a0 0x7ff27c0f55c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7ff278009fb0 tx=0x7ff278009b30).stop 2024-02-19T22:16:34.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:34.689+0000 7ff282995700 1 -- 172.21.15.88:0/1230779265 shutdown_connections 2024-02-19T22:16:34.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:34.689+0000 7ff282995700 1 --2- 172.21.15.88:0/1230779265 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff27c0f51a0 0x7ff27c0f55c0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:34.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:34.689+0000 7ff282995700 1 -- 172.21.15.88:0/1230779265 >> 172.21.15.88:0/1230779265 conn(0x7ff27c0f09a0 msgr2=0x7ff27c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:34.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:34.689+0000 7ff282995700 1 -- 172.21.15.88:0/1230779265 shutdown_connections 2024-02-19T22:16:34.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:34.689+0000 7ff282995700 1 -- 172.21.15.88:0/1230779265 wait complete. 2024-02-19T22:16:34.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:34.689+0000 7ff282995700 1 Processor -- start 2024-02-19T22:16:34.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:34.689+0000 7ff282995700 1 -- start start 2024-02-19T22:16:34.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:34.690+0000 7ff282995700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff27c0f51a0 0x7ff27c1054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:34.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:34.690+0000 7ff282995700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff27c1059c0 con 0x7ff27c0f51a0 2024-02-19T22:16:34.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:34.690+0000 7ff281993700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff27c0f51a0 0x7ff27c1054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:34.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:34.690+0000 7ff281993700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff27c0f51a0 0x7ff27c1054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:51740/0 (socket says 172.21.15.88:51740) 2024-02-19T22:16:34.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:34.690+0000 7ff281993700 1 -- 172.21.15.88:0/3752107815 learned_addr learned my addr 172.21.15.88:0/3752107815 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:16:34.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:34.690+0000 7ff281993700 1 -- 172.21.15.88:0/3752107815 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff278005040 con 0x7ff27c0f51a0 2024-02-19T22:16:34.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:34.690+0000 7ff281993700 1 --2- 172.21.15.88:0/3752107815 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff27c0f51a0 0x7ff27c1054d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7ff2780089f0 tx=0x7ff27800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:34.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:34.690+0000 7ff2727fc700 1 -- 172.21.15.88:0/3752107815 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff278009320 con 0x7ff27c0f51a0 2024-02-19T22:16:34.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:34.690+0000 7ff282995700 1 -- 172.21.15.88:0/3752107815 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff27c105bc0 con 0x7ff27c0f51a0 2024-02-19T22:16:34.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:34.690+0000 7ff2727fc700 1 -- 172.21.15.88:0/3752107815 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff278013380 con 0x7ff27c0f51a0 2024-02-19T22:16:34.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:34.690+0000 7ff2727fc700 1 -- 172.21.15.88:0/3752107815 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff278014e20 con 0x7ff27c0f51a0 2024-02-19T22:16:34.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:34.691+0000 7ff282995700 1 -- 172.21.15.88:0/3752107815 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff27c106050 con 0x7ff27c0f51a0 2024-02-19T22:16:34.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:34.691+0000 7ff282995700 1 -- 172.21.15.88:0/3752107815 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff27c045ad0 con 0x7ff27c0f51a0 2024-02-19T22:16:34.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:34.691+0000 7ff2727fc700 1 -- 172.21.15.88:0/3752107815 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff2780095f0 con 0x7ff27c0f51a0 2024-02-19T22:16:34.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:34.691+0000 7ff2727fc700 1 --2- 172.21.15.88:0/3752107815 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7ff26802ff10 0x7ff2680323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:34.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:34.691+0000 7ff2727fc700 1 -- 172.21.15.88:0/3752107815 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff278040ef0 con 0x7ff27c0f51a0 2024-02-19T22:16:34.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:34.692+0000 7ff281192700 1 --2- 172.21.15.88:0/3752107815 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7ff26802ff10 0x7ff2680323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:34.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:34.692+0000 7ff281192700 1 --2- 172.21.15.88:0/3752107815 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7ff26802ff10 0x7ff2680323d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7ff26c0096a0 tx=0x7ff26c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:34.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:34.696+0000 7ff2727fc700 1 -- 172.21.15.88:0/3752107815 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff278014420 con 0x7ff27c0f51a0 2024-02-19T22:16:34.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:34.899+0000 7ff282995700 1 -- 172.21.15.88:0/3752107815 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff27c106ea0 con 0x7ff27c0f51a0 2024-02-19T22:16:34.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:34.900+0000 7ff2727fc700 1 -- 172.21.15.88:0/3752107815 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff278010230 con 0x7ff27c0f51a0 2024-02-19T22:16:34.902 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:16:34.902 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:34.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:34.901+0000 7ff282995700 1 -- 172.21.15.88:0/3752107815 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7ff26802ff10 msgr2=0x7ff2680323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:34.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:34.901+0000 7ff282995700 1 --2- 172.21.15.88:0/3752107815 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7ff26802ff10 0x7ff2680323d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7ff26c0096a0 tx=0x7ff26c006b40).stop 2024-02-19T22:16:34.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:34.901+0000 7ff282995700 1 -- 172.21.15.88:0/3752107815 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff27c0f51a0 msgr2=0x7ff27c1054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:34.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:34.901+0000 7ff282995700 1 --2- 172.21.15.88:0/3752107815 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff27c0f51a0 0x7ff27c1054d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7ff2780089f0 tx=0x7ff27800b040).stop 2024-02-19T22:16:34.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:34.901+0000 7ff282995700 1 -- 172.21.15.88:0/3752107815 shutdown_connections 2024-02-19T22:16:34.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:34.901+0000 7ff282995700 1 --2- 172.21.15.88:0/3752107815 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7ff26802ff10 0x7ff2680323d0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:34.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:34.901+0000 7ff282995700 1 --2- 172.21.15.88:0/3752107815 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff27c0f51a0 0x7ff27c1054d0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:34.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:34.901+0000 7ff282995700 1 -- 172.21.15.88:0/3752107815 >> 172.21.15.88:0/3752107815 conn(0x7ff27c0f09a0 msgr2=0x7ff27c0f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:34.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:34.901+0000 7ff282995700 1 -- 172.21.15.88:0/3752107815 shutdown_connections 2024-02-19T22:16:34.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:34.901+0000 7ff282995700 1 -- 172.21.15.88:0/3752107815 wait complete. 2024-02-19T22:16:34.905 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:16:35.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:35 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:16:33.681620+0000 mgr.smithi052.ghqrtk (mgr.14162) 211 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:35.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:35 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:16:34.901255+0000 mon.smithi052 (mon.0) 239 : audit [DBG] from='client.? 172.21.15.88:0/3752107815' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:36.663 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:36.663 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:37.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:37 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:16:35.681979+0000 mgr.smithi052.ghqrtk (mgr.14162) 212 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:38.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:38.522+0000 7f1852b86700 1 -- 172.21.15.88:0/1850261376 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f184c0f5410 msgr2=0x7f184c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:38.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:38.522+0000 7f1852b86700 1 --2- 172.21.15.88:0/1850261376 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f184c0f5410 0x7f184c0f5830 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f1840009fb0 tx=0x7f1840009b30).stop 2024-02-19T22:16:38.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:38.522+0000 7f1852b86700 1 -- 172.21.15.88:0/1850261376 shutdown_connections 2024-02-19T22:16:38.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:38.522+0000 7f1852b86700 1 --2- 172.21.15.88:0/1850261376 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f184c0f5410 0x7f184c0f5830 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:38.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:38.522+0000 7f1852b86700 1 -- 172.21.15.88:0/1850261376 >> 172.21.15.88:0/1850261376 conn(0x7f184c0f0b70 msgr2=0x7f184c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:38.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:38.522+0000 7f1852b86700 1 -- 172.21.15.88:0/1850261376 shutdown_connections 2024-02-19T22:16:38.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:38.522+0000 7f1852b86700 1 -- 172.21.15.88:0/1850261376 wait complete. 2024-02-19T22:16:38.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:38.523+0000 7f1852b86700 1 Processor -- start 2024-02-19T22:16:38.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:38.523+0000 7f1852b86700 1 -- start start 2024-02-19T22:16:38.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:38.523+0000 7f1852b86700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f184c0f5410 0x7f184c1094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:38.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:38.523+0000 7f1852b86700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f184c060240 con 0x7f184c0f5410 2024-02-19T22:16:38.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:38.523+0000 7f1850922700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f184c0f5410 0x7f184c1094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:38.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:38.523+0000 7f1850922700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f184c0f5410 0x7f184c1094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:51772/0 (socket says 172.21.15.88:51772) 2024-02-19T22:16:38.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:38.523+0000 7f1850922700 1 -- 172.21.15.88:0/2866109710 learned_addr learned my addr 172.21.15.88:0/2866109710 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:16:38.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:38.524+0000 7f1850922700 1 -- 172.21.15.88:0/2866109710 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1840005040 con 0x7f184c0f5410 2024-02-19T22:16:38.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:38.524+0000 7f1850922700 1 --2- 172.21.15.88:0/2866109710 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f184c0f5410 0x7f184c1094e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f1840005010 tx=0x7f184000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:38.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:38.524+0000 7f18497fa700 1 -- 172.21.15.88:0/2866109710 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1840009320 con 0x7f184c0f5410 2024-02-19T22:16:38.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:38.524+0000 7f1852b86700 1 -- 172.21.15.88:0/2866109710 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f184c060440 con 0x7f184c0f5410 2024-02-19T22:16:38.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:38.524+0000 7f18497fa700 1 -- 172.21.15.88:0/2866109710 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1840013380 con 0x7f184c0f5410 2024-02-19T22:16:38.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:38.524+0000 7f18497fa700 1 -- 172.21.15.88:0/2866109710 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1840014e50 con 0x7f184c0f5410 2024-02-19T22:16:38.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:38.524+0000 7f1852b86700 1 -- 172.21.15.88:0/2866109710 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f184c109b90 con 0x7f184c0f5410 2024-02-19T22:16:38.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:38.525+0000 7f18497fa700 1 -- 172.21.15.88:0/2866109710 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f18400095f0 con 0x7f184c0f5410 2024-02-19T22:16:38.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:38.525+0000 7f1852b86700 1 -- 172.21.15.88:0/2866109710 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f184c045ad0 con 0x7f184c0f5410 2024-02-19T22:16:38.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:38.525+0000 7f18497fa700 1 --2- 172.21.15.88:0/2866109710 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f183c02ff10 0x7f183c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:38.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:38.525+0000 7f18497fa700 1 -- 172.21.15.88:0/2866109710 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1840040f70 con 0x7f184c0f5410 2024-02-19T22:16:38.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:38.525+0000 7f184bfff700 1 --2- 172.21.15.88:0/2866109710 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f183c02ff10 0x7f183c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:38.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:38.526+0000 7f184bfff700 1 --2- 172.21.15.88:0/2866109710 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f183c02ff10 0x7f183c0323d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f18380096a0 tx=0x7f1838006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:38.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:38.530+0000 7f18497fa700 1 -- 172.21.15.88:0/2866109710 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1840014420 con 0x7f184c0f5410 2024-02-19T22:16:38.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:38.736+0000 7f1852b86700 1 -- 172.21.15.88:0/2866109710 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f184c10a3a0 con 0x7f184c0f5410 2024-02-19T22:16:38.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:38.736+0000 7f18497fa700 1 -- 172.21.15.88:0/2866109710 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1840010230 con 0x7f184c0f5410 2024-02-19T22:16:38.739 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:16:38.739 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:38.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:38.738+0000 7f1852b86700 1 -- 172.21.15.88:0/2866109710 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f183c02ff10 msgr2=0x7f183c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:38.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:38.738+0000 7f1852b86700 1 --2- 172.21.15.88:0/2866109710 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f183c02ff10 0x7f183c0323d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f18380096a0 tx=0x7f1838006b40).stop 2024-02-19T22:16:38.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:38.738+0000 7f1852b86700 1 -- 172.21.15.88:0/2866109710 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f184c0f5410 msgr2=0x7f184c1094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:38.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:38.738+0000 7f1852b86700 1 --2- 172.21.15.88:0/2866109710 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f184c0f5410 0x7f184c1094e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f1840005010 tx=0x7f184000b040).stop 2024-02-19T22:16:38.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:38.738+0000 7f1852b86700 1 -- 172.21.15.88:0/2866109710 shutdown_connections 2024-02-19T22:16:38.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:38.738+0000 7f1852b86700 1 --2- 172.21.15.88:0/2866109710 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f183c02ff10 0x7f183c0323d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:38.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:38.738+0000 7f1852b86700 1 --2- 172.21.15.88:0/2866109710 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f184c0f5410 0x7f184c1094e0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:38.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:38.738+0000 7f1852b86700 1 -- 172.21.15.88:0/2866109710 >> 172.21.15.88:0/2866109710 conn(0x7f184c0f0b70 msgr2=0x7f184c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:38.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:38.738+0000 7f1852b86700 1 -- 172.21.15.88:0/2866109710 shutdown_connections 2024-02-19T22:16:38.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:38.738+0000 7f1852b86700 1 -- 172.21.15.88:0/2866109710 wait complete. 2024-02-19T22:16:38.741 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:16:39.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:16:37.682319+0000 mgr.smithi052.ghqrtk (mgr.14162) 213 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:39.844 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:16:38.737753+0000 mon.smithi052 (mon.0) 240 : audit [DBG] from='client.? 172.21.15.88:0/2866109710' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:40.543 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:40.543 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:41.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:41 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:16:39.682620+0000 mgr.smithi052.ghqrtk (mgr.14162) 214 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:42.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:42.510+0000 7f62f38df700 1 -- 172.21.15.88:0/2602174265 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62ec0f6510 msgr2=0x7f62ec0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:42.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:42.510+0000 7f62f38df700 1 --2- 172.21.15.88:0/2602174265 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62ec0f6510 0x7f62ec0f6930 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f62dc009fb0 tx=0x7f62dc009b30).stop 2024-02-19T22:16:42.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:42.510+0000 7f62f38df700 1 -- 172.21.15.88:0/2602174265 shutdown_connections 2024-02-19T22:16:42.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:42.510+0000 7f62f38df700 1 --2- 172.21.15.88:0/2602174265 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62ec0f6510 0x7f62ec0f6930 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:42.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:42.510+0000 7f62f38df700 1 -- 172.21.15.88:0/2602174265 >> 172.21.15.88:0/2602174265 conn(0x7f62ec0f1c70 msgr2=0x7f62ec0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:42.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:42.510+0000 7f62f38df700 1 -- 172.21.15.88:0/2602174265 shutdown_connections 2024-02-19T22:16:42.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:42.510+0000 7f62f38df700 1 -- 172.21.15.88:0/2602174265 wait complete. 2024-02-19T22:16:42.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:42.511+0000 7f62f38df700 1 Processor -- start 2024-02-19T22:16:42.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:42.511+0000 7f62f38df700 1 -- start start 2024-02-19T22:16:42.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:42.511+0000 7f62f38df700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62ec0f6510 0x7f62ec06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:42.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:42.511+0000 7f62f38df700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62ec069e20 con 0x7f62ec0f6510 2024-02-19T22:16:42.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:42.511+0000 7f62f167b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62ec0f6510 0x7f62ec06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:42.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:42.511+0000 7f62f167b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62ec0f6510 0x7f62ec06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:51794/0 (socket says 172.21.15.88:51794) 2024-02-19T22:16:42.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:42.511+0000 7f62f167b700 1 -- 172.21.15.88:0/1387559650 learned_addr learned my addr 172.21.15.88:0/1387559650 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:16:42.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:42.511+0000 7f62f167b700 1 -- 172.21.15.88:0/1387559650 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62dc005040 con 0x7f62ec0f6510 2024-02-19T22:16:42.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:42.511+0000 7f62f167b700 1 --2- 172.21.15.88:0/1387559650 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62ec0f6510 0x7f62ec06b9a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f62dc005010 tx=0x7f62dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:42.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:42.512+0000 7f62e1ffb700 1 -- 172.21.15.88:0/1387559650 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62dc009320 con 0x7f62ec0f6510 2024-02-19T22:16:42.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:42.512+0000 7f62f38df700 1 -- 172.21.15.88:0/1387559650 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62ec069fc0 con 0x7f62ec0f6510 2024-02-19T22:16:42.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:42.512+0000 7f62e1ffb700 1 -- 172.21.15.88:0/1387559650 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f62dc013380 con 0x7f62ec0f6510 2024-02-19T22:16:42.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:42.512+0000 7f62e1ffb700 1 -- 172.21.15.88:0/1387559650 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62dc014e50 con 0x7f62ec0f6510 2024-02-19T22:16:42.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:42.512+0000 7f62f38df700 1 -- 172.21.15.88:0/1387559650 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62ec06a450 con 0x7f62ec0f6510 2024-02-19T22:16:42.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:42.513+0000 7f62e1ffb700 1 -- 172.21.15.88:0/1387559650 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f62dc0095f0 con 0x7f62ec0f6510 2024-02-19T22:16:42.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:42.513+0000 7f62f38df700 1 -- 172.21.15.88:0/1387559650 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62ec059100 con 0x7f62ec0f6510 2024-02-19T22:16:42.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:42.513+0000 7f62e1ffb700 1 --2- 172.21.15.88:0/1387559650 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f62d802ff10 0x7f62d80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:42.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:42.513+0000 7f62e1ffb700 1 -- 172.21.15.88:0/1387559650 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f62dc040e90 con 0x7f62ec0f6510 2024-02-19T22:16:42.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:42.513+0000 7f62f0e7a700 1 --2- 172.21.15.88:0/1387559650 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f62d802ff10 0x7f62d80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:42.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:42.513+0000 7f62f0e7a700 1 --2- 172.21.15.88:0/1387559650 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f62d802ff10 0x7f62d80323d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f62e80096a0 tx=0x7f62e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:42.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:42.518+0000 7f62e1ffb700 1 -- 172.21.15.88:0/1387559650 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f62dc014420 con 0x7f62ec0f6510 2024-02-19T22:16:42.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:42.725+0000 7f62f38df700 1 -- 172.21.15.88:0/1387559650 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f62ec0fc0a0 con 0x7f62ec0f6510 2024-02-19T22:16:42.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:42.726+0000 7f62e1ffb700 1 -- 172.21.15.88:0/1387559650 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f62dc010230 con 0x7f62ec0f6510 2024-02-19T22:16:42.728 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:16:42.728 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:42.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:42.728+0000 7f62f38df700 1 -- 172.21.15.88:0/1387559650 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f62d802ff10 msgr2=0x7f62d80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:42.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:42.728+0000 7f62f38df700 1 --2- 172.21.15.88:0/1387559650 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f62d802ff10 0x7f62d80323d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f62e80096a0 tx=0x7f62e8006b40).stop 2024-02-19T22:16:42.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:42.728+0000 7f62f38df700 1 -- 172.21.15.88:0/1387559650 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62ec0f6510 msgr2=0x7f62ec06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:42.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:42.728+0000 7f62f38df700 1 --2- 172.21.15.88:0/1387559650 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62ec0f6510 0x7f62ec06b9a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f62dc005010 tx=0x7f62dc00b040).stop 2024-02-19T22:16:42.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:42.728+0000 7f62f38df700 1 -- 172.21.15.88:0/1387559650 shutdown_connections 2024-02-19T22:16:42.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:42.728+0000 7f62f38df700 1 --2- 172.21.15.88:0/1387559650 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f62d802ff10 0x7f62d80323d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:42.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:42.728+0000 7f62f38df700 1 --2- 172.21.15.88:0/1387559650 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62ec0f6510 0x7f62ec06b9a0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:42.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:42.728+0000 7f62f38df700 1 -- 172.21.15.88:0/1387559650 >> 172.21.15.88:0/1387559650 conn(0x7f62ec0f1c70 msgr2=0x7f62ec0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:42.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:42.728+0000 7f62f38df700 1 -- 172.21.15.88:0/1387559650 shutdown_connections 2024-02-19T22:16:42.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:42.728+0000 7f62f38df700 1 -- 172.21.15.88:0/1387559650 wait complete. 2024-02-19T22:16:42.731 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:16:43.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:43 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:16:41.682957+0000 mgr.smithi052.ghqrtk (mgr.14162) 215 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:43.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:43 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:16:42.727321+0000 mon.smithi052 (mon.0) 241 : audit [DBG] from='client.? 172.21.15.88:0/1387559650' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:44.500 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:44.500 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:44.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:44 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:16:43.751657+0000 mon.smithi052 (mon.0) 242 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi052.ghqrtk/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:16:44.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:44 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:16:43.754701+0000 mon.smithi052 (mon.0) 243 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi052.ghqrtk/trash_purge_schedule"}]: dispatch 2024-02-19T22:16:45.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:45 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:16:43.683283+0000 mgr.smithi052.ghqrtk (mgr.14162) 216 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:46.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:46.311+0000 7f1f852ea700 1 -- 172.21.15.88:0/3035831928 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1f800f6510 msgr2=0x7f1f800f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:46.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:46.311+0000 7f1f852ea700 1 --2- 172.21.15.88:0/3035831928 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1f800f6510 0x7f1f800f6930 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f1f78009fb0 tx=0x7f1f78009b30).stop 2024-02-19T22:16:46.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:46.311+0000 7f1f852ea700 1 -- 172.21.15.88:0/3035831928 shutdown_connections 2024-02-19T22:16:46.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:46.311+0000 7f1f852ea700 1 --2- 172.21.15.88:0/3035831928 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1f800f6510 0x7f1f800f6930 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:46.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:46.311+0000 7f1f852ea700 1 -- 172.21.15.88:0/3035831928 >> 172.21.15.88:0/3035831928 conn(0x7f1f800f1c70 msgr2=0x7f1f800f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:46.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:46.311+0000 7f1f852ea700 1 -- 172.21.15.88:0/3035831928 shutdown_connections 2024-02-19T22:16:46.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:46.311+0000 7f1f852ea700 1 -- 172.21.15.88:0/3035831928 wait complete. 2024-02-19T22:16:46.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:46.312+0000 7f1f852ea700 1 Processor -- start 2024-02-19T22:16:46.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:46.312+0000 7f1f852ea700 1 -- start start 2024-02-19T22:16:46.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:46.312+0000 7f1f852ea700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1f800f6510 0x7f1f801067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:46.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:46.312+0000 7f1f852ea700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f80106cd0 con 0x7f1f800f6510 2024-02-19T22:16:46.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:46.312+0000 7f1f7effd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1f800f6510 0x7f1f801067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:46.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:46.312+0000 7f1f7effd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1f800f6510 0x7f1f801067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:54480/0 (socket says 172.21.15.88:54480) 2024-02-19T22:16:46.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:46.312+0000 7f1f7effd700 1 -- 172.21.15.88:0/3238264475 learned_addr learned my addr 172.21.15.88:0/3238264475 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:16:46.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:46.312+0000 7f1f7effd700 1 -- 172.21.15.88:0/3238264475 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f78005040 con 0x7f1f800f6510 2024-02-19T22:16:46.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:46.313+0000 7f1f7effd700 1 --2- 172.21.15.88:0/3238264475 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1f800f6510 0x7f1f801067e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f1f780089f0 tx=0x7f1f7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:46.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:46.313+0000 7f1f777fe700 1 -- 172.21.15.88:0/3238264475 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f78009320 con 0x7f1f800f6510 2024-02-19T22:16:46.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:46.313+0000 7f1f852ea700 1 -- 172.21.15.88:0/3238264475 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f80106ed0 con 0x7f1f800f6510 2024-02-19T22:16:46.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:46.313+0000 7f1f777fe700 1 -- 172.21.15.88:0/3238264475 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1f78013380 con 0x7f1f800f6510 2024-02-19T22:16:46.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:46.313+0000 7f1f777fe700 1 -- 172.21.15.88:0/3238264475 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f7801ed80 con 0x7f1f800f6510 2024-02-19T22:16:46.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:46.313+0000 7f1f852ea700 1 -- 172.21.15.88:0/3238264475 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f80107360 con 0x7f1f800f6510 2024-02-19T22:16:46.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:46.314+0000 7f1f777fe700 1 -- 172.21.15.88:0/3238264475 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1f7801e420 con 0x7f1f800f6510 2024-02-19T22:16:46.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:46.314+0000 7f1f852ea700 1 -- 172.21.15.88:0/3238264475 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f80045ad0 con 0x7f1f800f6510 2024-02-19T22:16:46.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:46.314+0000 7f1f777fe700 1 --2- 172.21.15.88:0/3238264475 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f1f6c02ff60 0x7f1f6c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:46.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:46.314+0000 7f1f777fe700 1 -- 172.21.15.88:0/3238264475 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1f78041060 con 0x7f1f800f6510 2024-02-19T22:16:46.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:46.314+0000 7f1f7e7fc700 1 --2- 172.21.15.88:0/3238264475 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f1f6c02ff60 0x7f1f6c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:46.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:46.315+0000 7f1f7e7fc700 1 --2- 172.21.15.88:0/3238264475 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f1f6c02ff60 0x7f1f6c032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f1f680096a0 tx=0x7f1f68006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:46.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:46.319+0000 7f1f777fe700 1 -- 172.21.15.88:0/3238264475 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1f7801cb10 con 0x7f1f800f6510 2024-02-19T22:16:46.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:46.523+0000 7f1f852ea700 1 -- 172.21.15.88:0/3238264475 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1f801081b0 con 0x7f1f800f6510 2024-02-19T22:16:46.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:46.523+0000 7f1f777fe700 1 -- 172.21.15.88:0/3238264475 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1f7802c2d0 con 0x7f1f800f6510 2024-02-19T22:16:46.526 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:16:46.526 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:46.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:46.525+0000 7f1f852ea700 1 -- 172.21.15.88:0/3238264475 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f1f6c02ff60 msgr2=0x7f1f6c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:46.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:46.525+0000 7f1f852ea700 1 --2- 172.21.15.88:0/3238264475 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f1f6c02ff60 0x7f1f6c032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f1f680096a0 tx=0x7f1f68006b40).stop 2024-02-19T22:16:46.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:46.525+0000 7f1f852ea700 1 -- 172.21.15.88:0/3238264475 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1f800f6510 msgr2=0x7f1f801067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:46.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:46.525+0000 7f1f852ea700 1 --2- 172.21.15.88:0/3238264475 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1f800f6510 0x7f1f801067e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f1f780089f0 tx=0x7f1f7800b040).stop 2024-02-19T22:16:46.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:46.525+0000 7f1f852ea700 1 -- 172.21.15.88:0/3238264475 shutdown_connections 2024-02-19T22:16:46.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:46.525+0000 7f1f852ea700 1 --2- 172.21.15.88:0/3238264475 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f1f6c02ff60 0x7f1f6c032420 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:46.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:46.525+0000 7f1f852ea700 1 --2- 172.21.15.88:0/3238264475 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1f800f6510 0x7f1f801067e0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:46.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:46.525+0000 7f1f852ea700 1 -- 172.21.15.88:0/3238264475 >> 172.21.15.88:0/3238264475 conn(0x7f1f800f1c70 msgr2=0x7f1f800f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:46.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:46.525+0000 7f1f852ea700 1 -- 172.21.15.88:0/3238264475 shutdown_connections 2024-02-19T22:16:46.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:46.525+0000 7f1f852ea700 1 -- 172.21.15.88:0/3238264475 wait complete. 2024-02-19T22:16:46.528 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:16:47.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:47 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:16:45.683637+0000 mgr.smithi052.ghqrtk (mgr.14162) 217 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:47.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:47 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:16:46.524955+0000 mon.smithi052 (mon.0) 244 : audit [DBG] from='client.? 172.21.15.88:0/3238264475' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:48.259 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:48.259 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:49.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:49 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:16:47.684024+0000 mgr.smithi052.ghqrtk (mgr.14162) 218 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:50.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:50.167+0000 7f9ddd81d700 1 -- 172.21.15.88:0/3005894238 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9dd80f6510 msgr2=0x7f9dd80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:50.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:50.167+0000 7f9ddd81d700 1 --2- 172.21.15.88:0/3005894238 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9dd80f6510 0x7f9dd80f6930 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f9dc0009fb0 tx=0x7f9dc0009b30).stop 2024-02-19T22:16:50.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:50.167+0000 7f9ddd81d700 1 -- 172.21.15.88:0/3005894238 shutdown_connections 2024-02-19T22:16:50.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:50.167+0000 7f9ddd81d700 1 --2- 172.21.15.88:0/3005894238 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9dd80f6510 0x7f9dd80f6930 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:50.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:50.167+0000 7f9ddd81d700 1 -- 172.21.15.88:0/3005894238 >> 172.21.15.88:0/3005894238 conn(0x7f9dd80f1c70 msgr2=0x7f9dd80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:50.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:50.167+0000 7f9ddd81d700 1 -- 172.21.15.88:0/3005894238 shutdown_connections 2024-02-19T22:16:50.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:50.167+0000 7f9ddd81d700 1 -- 172.21.15.88:0/3005894238 wait complete. 2024-02-19T22:16:50.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:50.168+0000 7f9ddd81d700 1 Processor -- start 2024-02-19T22:16:50.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:50.168+0000 7f9ddd81d700 1 -- start start 2024-02-19T22:16:50.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:50.168+0000 7f9ddd81d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9dd80f6510 0x7f9dd81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:50.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:50.168+0000 7f9ddd81d700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd8106cd0 con 0x7f9dd80f6510 2024-02-19T22:16:50.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:50.169+0000 7f9dd6ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9dd80f6510 0x7f9dd81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:50.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:50.169+0000 7f9dd6ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9dd80f6510 0x7f9dd81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:54504/0 (socket says 172.21.15.88:54504) 2024-02-19T22:16:50.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:50.169+0000 7f9dd6ffd700 1 -- 172.21.15.88:0/3981658149 learned_addr learned my addr 172.21.15.88:0/3981658149 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:16:50.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:50.169+0000 7f9dd6ffd700 1 -- 172.21.15.88:0/3981658149 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dc0005040 con 0x7f9dd80f6510 2024-02-19T22:16:50.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:50.169+0000 7f9dd6ffd700 1 --2- 172.21.15.88:0/3981658149 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9dd80f6510 0x7f9dd81067e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f9dc0005010 tx=0x7f9dc000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:50.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:50.169+0000 7f9dcf7fe700 1 -- 172.21.15.88:0/3981658149 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9dc0009320 con 0x7f9dd80f6510 2024-02-19T22:16:50.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:50.169+0000 7f9dcf7fe700 1 -- 172.21.15.88:0/3981658149 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9dc0013380 con 0x7f9dd80f6510 2024-02-19T22:16:50.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:50.169+0000 7f9ddd81d700 1 -- 172.21.15.88:0/3981658149 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dd8106ed0 con 0x7f9dd80f6510 2024-02-19T22:16:50.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:50.169+0000 7f9dcf7fe700 1 -- 172.21.15.88:0/3981658149 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9dc0014e50 con 0x7f9dd80f6510 2024-02-19T22:16:50.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:50.169+0000 7f9ddd81d700 1 -- 172.21.15.88:0/3981658149 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dd8107360 con 0x7f9dd80f6510 2024-02-19T22:16:50.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:50.170+0000 7f9dcf7fe700 1 -- 172.21.15.88:0/3981658149 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9dc00263a0 con 0x7f9dd80f6510 2024-02-19T22:16:50.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:50.170+0000 7f9dcf7fe700 1 --2- 172.21.15.88:0/3981658149 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f9dc402ff10 0x7f9dc40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:50.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:50.170+0000 7f9dcf7fe700 1 -- 172.21.15.88:0/3981658149 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9dc0040fa0 con 0x7f9dd80f6510 2024-02-19T22:16:50.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:50.170+0000 7f9ddd81d700 1 -- 172.21.15.88:0/3981658149 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dd8045ad0 con 0x7f9dd80f6510 2024-02-19T22:16:50.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:50.170+0000 7f9dd67fc700 1 --2- 172.21.15.88:0/3981658149 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f9dc402ff10 0x7f9dc40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:50.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:50.171+0000 7f9dd67fc700 1 --2- 172.21.15.88:0/3981658149 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f9dc402ff10 0x7f9dc40323d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f9dc80096a0 tx=0x7f9dc8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:50.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:50.175+0000 7f9dcf7fe700 1 -- 172.21.15.88:0/3981658149 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9dc0012920 con 0x7f9dd80f6510 2024-02-19T22:16:50.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:50.380+0000 7f9ddd81d700 1 -- 172.21.15.88:0/3981658149 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9dd8108210 con 0x7f9dd80f6510 2024-02-19T22:16:50.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:50.382+0000 7f9dcf7fe700 1 -- 172.21.15.88:0/3981658149 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9dc002b020 con 0x7f9dd80f6510 2024-02-19T22:16:50.384 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:16:50.384 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:50.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:50.383+0000 7f9ddd81d700 1 -- 172.21.15.88:0/3981658149 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f9dc402ff10 msgr2=0x7f9dc40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:50.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:50.383+0000 7f9ddd81d700 1 --2- 172.21.15.88:0/3981658149 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f9dc402ff10 0x7f9dc40323d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f9dc80096a0 tx=0x7f9dc8006b40).stop 2024-02-19T22:16:50.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:50.383+0000 7f9ddd81d700 1 -- 172.21.15.88:0/3981658149 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9dd80f6510 msgr2=0x7f9dd81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:50.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:50.383+0000 7f9ddd81d700 1 --2- 172.21.15.88:0/3981658149 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9dd80f6510 0x7f9dd81067e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f9dc0005010 tx=0x7f9dc000b040).stop 2024-02-19T22:16:50.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:50.383+0000 7f9ddd81d700 1 -- 172.21.15.88:0/3981658149 shutdown_connections 2024-02-19T22:16:50.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:50.383+0000 7f9ddd81d700 1 --2- 172.21.15.88:0/3981658149 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f9dc402ff10 0x7f9dc40323d0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:50.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:50.383+0000 7f9ddd81d700 1 --2- 172.21.15.88:0/3981658149 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9dd80f6510 0x7f9dd81067e0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:50.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:50.383+0000 7f9ddd81d700 1 -- 172.21.15.88:0/3981658149 >> 172.21.15.88:0/3981658149 conn(0x7f9dd80f1c70 msgr2=0x7f9dd80f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:50.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:50.383+0000 7f9ddd81d700 1 -- 172.21.15.88:0/3981658149 shutdown_connections 2024-02-19T22:16:50.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:50.383+0000 7f9ddd81d700 1 -- 172.21.15.88:0/3981658149 wait complete. 2024-02-19T22:16:50.386 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:16:50.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:50 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:16:50.382474+0000 mon.smithi052 (mon.0) 245 : audit [DBG] from='client.? 172.21.15.88:0/3981658149' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:51.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:51 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:16:49.684430+0000 mgr.smithi052.ghqrtk (mgr.14162) 219 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:51.987 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:51.987 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:53.843 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:53 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:16:51.684755+0000 mgr.smithi052.ghqrtk (mgr.14162) 220 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:53.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:53.937+0000 7f52b9f46700 1 -- 172.21.15.88:0/4081028823 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f52b40f5410 msgr2=0x7f52b40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:53.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:53.937+0000 7f52b9f46700 1 --2- 172.21.15.88:0/4081028823 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f52b40f5410 0x7f52b40f5830 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f52a8009fb0 tx=0x7f52a8009b30).stop 2024-02-19T22:16:53.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:53.937+0000 7f52b9f46700 1 -- 172.21.15.88:0/4081028823 shutdown_connections 2024-02-19T22:16:53.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:53.937+0000 7f52b9f46700 1 --2- 172.21.15.88:0/4081028823 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f52b40f5410 0x7f52b40f5830 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:53.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:53.937+0000 7f52b9f46700 1 -- 172.21.15.88:0/4081028823 >> 172.21.15.88:0/4081028823 conn(0x7f52b40f0b70 msgr2=0x7f52b40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:53.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:53.937+0000 7f52b9f46700 1 -- 172.21.15.88:0/4081028823 shutdown_connections 2024-02-19T22:16:53.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:53.937+0000 7f52b9f46700 1 -- 172.21.15.88:0/4081028823 wait complete. 2024-02-19T22:16:53.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:53.938+0000 7f52b9f46700 1 Processor -- start 2024-02-19T22:16:53.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:53.938+0000 7f52b9f46700 1 -- start start 2024-02-19T22:16:53.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:53.938+0000 7f52b9f46700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f52b40f5410 0x7f52b41056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:53.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:53.938+0000 7f52b9f46700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b4105bd0 con 0x7f52b40f5410 2024-02-19T22:16:53.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:53.938+0000 7f52b37fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f52b40f5410 0x7f52b41056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:53.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:53.938+0000 7f52b37fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f52b40f5410 0x7f52b41056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:55710/0 (socket says 172.21.15.88:55710) 2024-02-19T22:16:53.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:53.938+0000 7f52b37fe700 1 -- 172.21.15.88:0/1289450282 learned_addr learned my addr 172.21.15.88:0/1289450282 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:16:53.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:53.939+0000 7f52b37fe700 1 -- 172.21.15.88:0/1289450282 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a8005040 con 0x7f52b40f5410 2024-02-19T22:16:53.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:53.939+0000 7f52b37fe700 1 --2- 172.21.15.88:0/1289450282 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f52b40f5410 0x7f52b41056e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f52a8005010 tx=0x7f52a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:53.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:53.939+0000 7f529bfff700 1 -- 172.21.15.88:0/1289450282 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52a8009320 con 0x7f52b40f5410 2024-02-19T22:16:53.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:53.939+0000 7f52b9f46700 1 -- 172.21.15.88:0/1289450282 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52b4105dd0 con 0x7f52b40f5410 2024-02-19T22:16:53.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:53.939+0000 7f529bfff700 1 -- 172.21.15.88:0/1289450282 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f52a8013380 con 0x7f52b40f5410 2024-02-19T22:16:53.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:53.939+0000 7f529bfff700 1 -- 172.21.15.88:0/1289450282 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52a8014e50 con 0x7f52b40f5410 2024-02-19T22:16:53.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:53.939+0000 7f52b9f46700 1 -- 172.21.15.88:0/1289450282 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52b4106260 con 0x7f52b40f5410 2024-02-19T22:16:53.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:53.940+0000 7f529bfff700 1 -- 172.21.15.88:0/1289450282 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f52a80095f0 con 0x7f52b40f5410 2024-02-19T22:16:53.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:53.940+0000 7f52b9f46700 1 -- 172.21.15.88:0/1289450282 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52b4045ad0 con 0x7f52b40f5410 2024-02-19T22:16:53.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:53.940+0000 7f529bfff700 1 --2- 172.21.15.88:0/1289450282 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f529c02ff10 0x7f529c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:53.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:53.940+0000 7f529bfff700 1 -- 172.21.15.88:0/1289450282 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f52a8040e70 con 0x7f52b40f5410 2024-02-19T22:16:53.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:53.940+0000 7f52b2ffd700 1 --2- 172.21.15.88:0/1289450282 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f529c02ff10 0x7f529c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:53.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:53.941+0000 7f52b2ffd700 1 --2- 172.21.15.88:0/1289450282 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f529c02ff10 0x7f529c0323d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f52a40096a0 tx=0x7f52a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:53.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:53.945+0000 7f529bfff700 1 -- 172.21.15.88:0/1289450282 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f52a8014420 con 0x7f52b40f5410 2024-02-19T22:16:54.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:54.150+0000 7f52b9f46700 1 -- 172.21.15.88:0/1289450282 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f52b4107110 con 0x7f52b40f5410 2024-02-19T22:16:54.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:54.150+0000 7f529bfff700 1 -- 172.21.15.88:0/1289450282 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f52a8010230 con 0x7f52b40f5410 2024-02-19T22:16:54.153 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:16:54.153 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:54.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:54.152+0000 7f52b9f46700 1 -- 172.21.15.88:0/1289450282 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f529c02ff10 msgr2=0x7f529c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:54.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:54.152+0000 7f52b9f46700 1 --2- 172.21.15.88:0/1289450282 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f529c02ff10 0x7f529c0323d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f52a40096a0 tx=0x7f52a4006b40).stop 2024-02-19T22:16:54.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:54.152+0000 7f52b9f46700 1 -- 172.21.15.88:0/1289450282 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f52b40f5410 msgr2=0x7f52b41056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:54.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:54.152+0000 7f52b9f46700 1 --2- 172.21.15.88:0/1289450282 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f52b40f5410 0x7f52b41056e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f52a8005010 tx=0x7f52a800b040).stop 2024-02-19T22:16:54.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:54.152+0000 7f52b9f46700 1 -- 172.21.15.88:0/1289450282 shutdown_connections 2024-02-19T22:16:54.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:54.152+0000 7f52b9f46700 1 --2- 172.21.15.88:0/1289450282 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f529c02ff10 0x7f529c0323d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:54.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:54.152+0000 7f52b9f46700 1 --2- 172.21.15.88:0/1289450282 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f52b40f5410 0x7f52b41056e0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:54.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:54.152+0000 7f52b9f46700 1 -- 172.21.15.88:0/1289450282 >> 172.21.15.88:0/1289450282 conn(0x7f52b40f0b70 msgr2=0x7f52b40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:54.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:54.152+0000 7f52b9f46700 1 -- 172.21.15.88:0/1289450282 shutdown_connections 2024-02-19T22:16:54.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:54.152+0000 7f52b9f46700 1 -- 172.21.15.88:0/1289450282 wait complete. 2024-02-19T22:16:54.155 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:16:55.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:55 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:16:53.685100+0000 mgr.smithi052.ghqrtk (mgr.14162) 221 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:55.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:55 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:16:54.151768+0000 mon.smithi052 (mon.0) 246 : audit [DBG] from='client.? 172.21.15.88:0/1289450282' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:55.881 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:55.899 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:57.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:57 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:16:55.685503+0000 mgr.smithi052.ghqrtk (mgr.14162) 222 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:57.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:57.842+0000 7eff7ed00700 1 -- 172.21.15.88:0/3784875667 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7eff780f5410 msgr2=0x7eff780f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:57.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:57.842+0000 7eff7ed00700 1 --2- 172.21.15.88:0/3784875667 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7eff780f5410 0x7eff780f5830 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7eff6c009fb0 tx=0x7eff6c009b30).stop 2024-02-19T22:16:57.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:57.842+0000 7eff7ed00700 1 -- 172.21.15.88:0/3784875667 shutdown_connections 2024-02-19T22:16:57.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:57.842+0000 7eff7ed00700 1 --2- 172.21.15.88:0/3784875667 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7eff780f5410 0x7eff780f5830 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:57.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:57.842+0000 7eff7ed00700 1 -- 172.21.15.88:0/3784875667 >> 172.21.15.88:0/3784875667 conn(0x7eff780f0b70 msgr2=0x7eff780f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:57.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:57.842+0000 7eff7ed00700 1 -- 172.21.15.88:0/3784875667 shutdown_connections 2024-02-19T22:16:57.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:57.842+0000 7eff7ed00700 1 -- 172.21.15.88:0/3784875667 wait complete. 2024-02-19T22:16:57.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:57.842+0000 7eff7ed00700 1 Processor -- start 2024-02-19T22:16:57.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:57.842+0000 7eff7ed00700 1 -- start start 2024-02-19T22:16:57.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:57.843+0000 7eff7ed00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7eff780f5410 0x7eff781056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:57.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:57.843+0000 7eff7ed00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff78105bd0 con 0x7eff780f5410 2024-02-19T22:16:57.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:57.843+0000 7eff7ca9c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7eff780f5410 0x7eff781056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:57.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:57.843+0000 7eff7ca9c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7eff780f5410 0x7eff781056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:55720/0 (socket says 172.21.15.88:55720) 2024-02-19T22:16:57.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:57.843+0000 7eff7ca9c700 1 -- 172.21.15.88:0/1737861883 learned_addr learned my addr 172.21.15.88:0/1737861883 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:16:57.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:57.843+0000 7eff7ca9c700 1 -- 172.21.15.88:0/1737861883 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff6c005040 con 0x7eff780f5410 2024-02-19T22:16:57.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:57.843+0000 7eff7ca9c700 1 --2- 172.21.15.88:0/1737861883 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7eff780f5410 0x7eff781056e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7eff6c005010 tx=0x7eff6c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:57.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:57.843+0000 7eff757fa700 1 -- 172.21.15.88:0/1737861883 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff6c009320 con 0x7eff780f5410 2024-02-19T22:16:57.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:57.843+0000 7eff7ed00700 1 -- 172.21.15.88:0/1737861883 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff78105dd0 con 0x7eff780f5410 2024-02-19T22:16:57.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:57.843+0000 7eff757fa700 1 -- 172.21.15.88:0/1737861883 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7eff6c013380 con 0x7eff780f5410 2024-02-19T22:16:57.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:57.843+0000 7eff757fa700 1 -- 172.21.15.88:0/1737861883 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff6c014e50 con 0x7eff780f5410 2024-02-19T22:16:57.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:57.844+0000 7eff7ed00700 1 -- 172.21.15.88:0/1737861883 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff78106260 con 0x7eff780f5410 2024-02-19T22:16:57.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:57.844+0000 7eff757fa700 1 -- 172.21.15.88:0/1737861883 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7eff6c0095f0 con 0x7eff780f5410 2024-02-19T22:16:57.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:57.844+0000 7eff7ed00700 1 -- 172.21.15.88:0/1737861883 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff78045ad0 con 0x7eff780f5410 2024-02-19T22:16:57.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:57.844+0000 7eff757fa700 1 --2- 172.21.15.88:0/1737861883 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7eff6802ff10 0x7eff680323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:57.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:57.844+0000 7eff757fa700 1 -- 172.21.15.88:0/1737861883 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7eff6c040e70 con 0x7eff780f5410 2024-02-19T22:16:57.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:57.845+0000 7eff77fff700 1 --2- 172.21.15.88:0/1737861883 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7eff6802ff10 0x7eff680323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:57.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:57.845+0000 7eff77fff700 1 --2- 172.21.15.88:0/1737861883 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7eff6802ff10 0x7eff680323d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7eff640096a0 tx=0x7eff64006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:57.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:57.850+0000 7eff757fa700 1 -- 172.21.15.88:0/1737861883 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7eff6c014420 con 0x7eff780f5410 2024-02-19T22:16:58.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:58.053+0000 7eff7ed00700 1 -- 172.21.15.88:0/1737861883 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7eff781070b0 con 0x7eff780f5410 2024-02-19T22:16:58.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:58.055+0000 7eff757fa700 1 -- 172.21.15.88:0/1737861883 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7eff6c010230 con 0x7eff780f5410 2024-02-19T22:16:58.057 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:16:58.057 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:58.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:58.055+0000 7eff7ed00700 1 -- 172.21.15.88:0/1737861883 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7eff6802ff10 msgr2=0x7eff680323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:58.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:58.055+0000 7eff7ed00700 1 --2- 172.21.15.88:0/1737861883 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7eff6802ff10 0x7eff680323d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7eff640096a0 tx=0x7eff64006b40).stop 2024-02-19T22:16:58.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:58.056+0000 7eff7ed00700 1 -- 172.21.15.88:0/1737861883 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7eff780f5410 msgr2=0x7eff781056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:58.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:58.056+0000 7eff7ed00700 1 --2- 172.21.15.88:0/1737861883 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7eff780f5410 0x7eff781056e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7eff6c005010 tx=0x7eff6c00b040).stop 2024-02-19T22:16:58.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:58.056+0000 7eff7ed00700 1 -- 172.21.15.88:0/1737861883 shutdown_connections 2024-02-19T22:16:58.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:58.056+0000 7eff7ed00700 1 --2- 172.21.15.88:0/1737861883 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7eff6802ff10 0x7eff680323d0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:58.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:58.056+0000 7eff7ed00700 1 --2- 172.21.15.88:0/1737861883 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7eff780f5410 0x7eff781056e0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:58.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:58.056+0000 7eff7ed00700 1 -- 172.21.15.88:0/1737861883 >> 172.21.15.88:0/1737861883 conn(0x7eff780f0b70 msgr2=0x7eff780f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:58.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:58.056+0000 7eff7ed00700 1 -- 172.21.15.88:0/1737861883 shutdown_connections 2024-02-19T22:16:58.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:16:58.056+0000 7eff7ed00700 1 -- 172.21.15.88:0/1737861883 wait complete. 2024-02-19T22:16:58.059 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:16:58.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:58 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:16:58.055481+0000 mon.smithi052 (mon.0) 247 : audit [DBG] from='client.? 172.21.15.88:0/1737861883' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:59.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:16:59 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:16:57.685915+0000 mgr.smithi052.ghqrtk (mgr.14162) 223 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:59.756 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:59.756 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:01.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:01 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:16:59.686315+0000 mgr.smithi052.ghqrtk (mgr.14162) 224 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:01.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:01.616+0000 7fb4761f3700 1 -- 172.21.15.88:0/3678561255 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb4700f5410 msgr2=0x7fb4700f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:01.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:01.616+0000 7fb4761f3700 1 --2- 172.21.15.88:0/3678561255 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb4700f5410 0x7fb4700f5830 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7fb464009fb0 tx=0x7fb464009b30).stop 2024-02-19T22:17:01.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:01.616+0000 7fb4761f3700 1 -- 172.21.15.88:0/3678561255 shutdown_connections 2024-02-19T22:17:01.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:01.616+0000 7fb4761f3700 1 --2- 172.21.15.88:0/3678561255 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb4700f5410 0x7fb4700f5830 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:01.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:01.616+0000 7fb4761f3700 1 -- 172.21.15.88:0/3678561255 >> 172.21.15.88:0/3678561255 conn(0x7fb4700f0b70 msgr2=0x7fb4700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:01.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:01.616+0000 7fb4761f3700 1 -- 172.21.15.88:0/3678561255 shutdown_connections 2024-02-19T22:17:01.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:01.616+0000 7fb4761f3700 1 -- 172.21.15.88:0/3678561255 wait complete. 2024-02-19T22:17:01.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:01.617+0000 7fb4761f3700 1 Processor -- start 2024-02-19T22:17:01.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:01.617+0000 7fb4761f3700 1 -- start start 2024-02-19T22:17:01.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:01.617+0000 7fb4761f3700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb4700f5410 0x7fb4700fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:01.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:01.617+0000 7fb4761f3700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4700fb960 con 0x7fb4700f5410 2024-02-19T22:17:01.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:01.617+0000 7fb46f7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb4700f5410 0x7fb4700fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:01.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:01.617+0000 7fb46f7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb4700f5410 0x7fb4700fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:55736/0 (socket says 172.21.15.88:55736) 2024-02-19T22:17:01.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:01.617+0000 7fb46f7fe700 1 -- 172.21.15.88:0/544598000 learned_addr learned my addr 172.21.15.88:0/544598000 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:17:01.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:01.618+0000 7fb46f7fe700 1 -- 172.21.15.88:0/544598000 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb464005040 con 0x7fb4700f5410 2024-02-19T22:17:01.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:01.618+0000 7fb46f7fe700 1 --2- 172.21.15.88:0/544598000 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb4700f5410 0x7fb4700fd4e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fb464005010 tx=0x7fb46400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:01.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:01.618+0000 7fb457fff700 1 -- 172.21.15.88:0/544598000 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb464009320 con 0x7fb4700f5410 2024-02-19T22:17:01.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:01.618+0000 7fb4761f3700 1 -- 172.21.15.88:0/544598000 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4700fbb00 con 0x7fb4700f5410 2024-02-19T22:17:01.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:01.618+0000 7fb457fff700 1 -- 172.21.15.88:0/544598000 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb464013380 con 0x7fb4700f5410 2024-02-19T22:17:01.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:01.618+0000 7fb457fff700 1 -- 172.21.15.88:0/544598000 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb464014e50 con 0x7fb4700f5410 2024-02-19T22:17:01.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:01.618+0000 7fb4761f3700 1 -- 172.21.15.88:0/544598000 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4700fbf90 con 0x7fb4700f5410 2024-02-19T22:17:01.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:01.619+0000 7fb457fff700 1 -- 172.21.15.88:0/544598000 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb4640095f0 con 0x7fb4700f5410 2024-02-19T22:17:01.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:01.619+0000 7fb4761f3700 1 -- 172.21.15.88:0/544598000 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb470059100 con 0x7fb4700f5410 2024-02-19T22:17:01.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:01.619+0000 7fb457fff700 1 --2- 172.21.15.88:0/544598000 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fb45802ff10 0x7fb4580323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:01.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:01.619+0000 7fb457fff700 1 -- 172.21.15.88:0/544598000 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb464040e60 con 0x7fb4700f5410 2024-02-19T22:17:01.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:01.619+0000 7fb46effd700 1 --2- 172.21.15.88:0/544598000 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fb45802ff10 0x7fb4580323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:01.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:01.620+0000 7fb46effd700 1 --2- 172.21.15.88:0/544598000 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fb45802ff10 0x7fb4580323d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fb4600096a0 tx=0x7fb460006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:01.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:01.624+0000 7fb457fff700 1 -- 172.21.15.88:0/544598000 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb464014420 con 0x7fb4700f5410 2024-02-19T22:17:01.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:01.826+0000 7fb4761f3700 1 -- 172.21.15.88:0/544598000 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb470045ad0 con 0x7fb4700f5410 2024-02-19T22:17:01.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:01.827+0000 7fb457fff700 1 -- 172.21.15.88:0/544598000 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb464010230 con 0x7fb4700f5410 2024-02-19T22:17:01.828 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:17:01.828 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:01.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:01.828+0000 7fb4761f3700 1 -- 172.21.15.88:0/544598000 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fb45802ff10 msgr2=0x7fb4580323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:01.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:01.828+0000 7fb4761f3700 1 --2- 172.21.15.88:0/544598000 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fb45802ff10 0x7fb4580323d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fb4600096a0 tx=0x7fb460006b40).stop 2024-02-19T22:17:01.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:01.828+0000 7fb4761f3700 1 -- 172.21.15.88:0/544598000 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb4700f5410 msgr2=0x7fb4700fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:01.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:01.828+0000 7fb4761f3700 1 --2- 172.21.15.88:0/544598000 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb4700f5410 0x7fb4700fd4e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fb464005010 tx=0x7fb46400b040).stop 2024-02-19T22:17:01.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:01.829+0000 7fb4761f3700 1 -- 172.21.15.88:0/544598000 shutdown_connections 2024-02-19T22:17:01.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:01.829+0000 7fb4761f3700 1 --2- 172.21.15.88:0/544598000 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fb45802ff10 0x7fb4580323d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:01.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:01.829+0000 7fb4761f3700 1 --2- 172.21.15.88:0/544598000 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb4700f5410 0x7fb4700fd4e0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:01.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:01.829+0000 7fb4761f3700 1 -- 172.21.15.88:0/544598000 >> 172.21.15.88:0/544598000 conn(0x7fb4700f0b70 msgr2=0x7fb47005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:01.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:01.829+0000 7fb4761f3700 1 -- 172.21.15.88:0/544598000 shutdown_connections 2024-02-19T22:17:01.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:01.829+0000 7fb4761f3700 1 -- 172.21.15.88:0/544598000 wait complete. 2024-02-19T22:17:01.831 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:17:02.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:02 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:17:01.827427+0000 mon.smithi052 (mon.0) 248 : audit [DBG] from='client.? 172.21.15.88:0/544598000' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:03.522 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:03.523 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:03.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:03 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:17:01.686597+0000 mgr.smithi052.ghqrtk (mgr.14162) 225 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:05.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:05.430+0000 7f9b5e4ff700 1 -- 172.21.15.88:0/3669361648 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b580f51a0 msgr2=0x7f9b580f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:05.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:05.430+0000 7f9b5e4ff700 1 --2- 172.21.15.88:0/3669361648 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b580f51a0 0x7f9b580f55c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f9b54009fb0 tx=0x7f9b54009b30).stop 2024-02-19T22:17:05.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:05.430+0000 7f9b5e4ff700 1 -- 172.21.15.88:0/3669361648 shutdown_connections 2024-02-19T22:17:05.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:05.430+0000 7f9b5e4ff700 1 --2- 172.21.15.88:0/3669361648 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b580f51a0 0x7f9b580f55c0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:05.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:05.430+0000 7f9b5e4ff700 1 -- 172.21.15.88:0/3669361648 >> 172.21.15.88:0/3669361648 conn(0x7f9b580f09a0 msgr2=0x7f9b580f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:05.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:05.430+0000 7f9b5e4ff700 1 -- 172.21.15.88:0/3669361648 shutdown_connections 2024-02-19T22:17:05.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:05.430+0000 7f9b5e4ff700 1 -- 172.21.15.88:0/3669361648 wait complete. 2024-02-19T22:17:05.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:05.430+0000 7f9b5e4ff700 1 Processor -- start 2024-02-19T22:17:05.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:05.430+0000 7f9b5e4ff700 1 -- start start 2024-02-19T22:17:05.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:05.431+0000 7f9b5e4ff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b580f51a0 0x7f9b5810b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:05.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:05.431+0000 7f9b5e4ff700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b581070e0 con 0x7f9b580f51a0 2024-02-19T22:17:05.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:05.431+0000 7f9b5d4fd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b580f51a0 0x7f9b5810b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:05.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:05.431+0000 7f9b5d4fd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b580f51a0 0x7f9b5810b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:35744/0 (socket says 172.21.15.88:35744) 2024-02-19T22:17:05.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:05.431+0000 7f9b5d4fd700 1 -- 172.21.15.88:0/3813775176 learned_addr learned my addr 172.21.15.88:0/3813775176 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:17:05.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:05.431+0000 7f9b5d4fd700 1 -- 172.21.15.88:0/3813775176 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b54005040 con 0x7f9b580f51a0 2024-02-19T22:17:05.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:05.431+0000 7f9b5d4fd700 1 --2- 172.21.15.88:0/3813775176 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b580f51a0 0x7f9b5810b6f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f9b540089f0 tx=0x7f9b5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:05.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:05.431+0000 7f9b4dffb700 1 -- 172.21.15.88:0/3813775176 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b54009320 con 0x7f9b580f51a0 2024-02-19T22:17:05.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:05.432+0000 7f9b5e4ff700 1 -- 172.21.15.88:0/3813775176 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b58060220 con 0x7f9b580f51a0 2024-02-19T22:17:05.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:05.432+0000 7f9b5e4ff700 1 -- 172.21.15.88:0/3813775176 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b5810bbe0 con 0x7f9b580f51a0 2024-02-19T22:17:05.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:05.432+0000 7f9b4dffb700 1 -- 172.21.15.88:0/3813775176 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9b54013380 con 0x7f9b580f51a0 2024-02-19T22:17:05.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:05.432+0000 7f9b4dffb700 1 -- 172.21.15.88:0/3813775176 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b54014e40 con 0x7f9b580f51a0 2024-02-19T22:17:05.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:05.432+0000 7f9b4dffb700 1 -- 172.21.15.88:0/3813775176 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9b54014420 con 0x7f9b580f51a0 2024-02-19T22:17:05.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:05.433+0000 7f9b5e4ff700 1 -- 172.21.15.88:0/3813775176 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b38004fa0 con 0x7f9b580f51a0 2024-02-19T22:17:05.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:05.433+0000 7f9b4dffb700 1 --2- 172.21.15.88:0/3813775176 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f9b4402ff10 0x7f9b440323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:05.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:05.433+0000 7f9b4dffb700 1 -- 172.21.15.88:0/3813775176 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9b54040ec0 con 0x7f9b580f51a0 2024-02-19T22:17:05.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:05.433+0000 7f9b5ccfc700 1 --2- 172.21.15.88:0/3813775176 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f9b4402ff10 0x7f9b440323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:05.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:05.433+0000 7f9b5ccfc700 1 --2- 172.21.15.88:0/3813775176 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f9b4402ff10 0x7f9b440323d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f9b480096a0 tx=0x7f9b48006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:05.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:05.438+0000 7f9b4dffb700 1 -- 172.21.15.88:0/3813775176 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9b54012bb0 con 0x7f9b580f51a0 2024-02-19T22:17:05.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:05 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:17:03.686887+0000 mgr.smithi052.ghqrtk (mgr.14162) 226 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:05.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:05.644+0000 7f9b5e4ff700 1 -- 172.21.15.88:0/3813775176 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9b38005670 con 0x7f9b580f51a0 2024-02-19T22:17:05.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:05.644+0000 7f9b4dffb700 1 -- 172.21.15.88:0/3813775176 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9b540127a0 con 0x7f9b580f51a0 2024-02-19T22:17:05.646 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:17:05.646 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:05.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:05.646+0000 7f9b5e4ff700 1 -- 172.21.15.88:0/3813775176 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f9b4402ff10 msgr2=0x7f9b440323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:05.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:05.646+0000 7f9b5e4ff700 1 --2- 172.21.15.88:0/3813775176 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f9b4402ff10 0x7f9b440323d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f9b480096a0 tx=0x7f9b48006b40).stop 2024-02-19T22:17:05.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:05.646+0000 7f9b5e4ff700 1 -- 172.21.15.88:0/3813775176 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b580f51a0 msgr2=0x7f9b5810b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:05.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:05.646+0000 7f9b5e4ff700 1 --2- 172.21.15.88:0/3813775176 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b580f51a0 0x7f9b5810b6f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f9b540089f0 tx=0x7f9b5400b040).stop 2024-02-19T22:17:05.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:05.646+0000 7f9b5e4ff700 1 -- 172.21.15.88:0/3813775176 shutdown_connections 2024-02-19T22:17:05.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:05.646+0000 7f9b5e4ff700 1 --2- 172.21.15.88:0/3813775176 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f9b4402ff10 0x7f9b440323d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:05.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:05.646+0000 7f9b5e4ff700 1 --2- 172.21.15.88:0/3813775176 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b580f51a0 0x7f9b5810b6f0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:05.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:05.646+0000 7f9b5e4ff700 1 -- 172.21.15.88:0/3813775176 >> 172.21.15.88:0/3813775176 conn(0x7f9b580f09a0 msgr2=0x7f9b580f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:05.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:05.646+0000 7f9b5e4ff700 1 -- 172.21.15.88:0/3813775176 shutdown_connections 2024-02-19T22:17:05.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:05.646+0000 7f9b5e4ff700 1 -- 172.21.15.88:0/3813775176 wait complete. 2024-02-19T22:17:05.648 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:17:06.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:06 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:17:05.644710+0000 mon.smithi052 (mon.0) 249 : audit [DBG] from='client.? 172.21.15.88:0/3813775176' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:07.334 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:07.334 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:07.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:07 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:17:05.687179+0000 mgr.smithi052.ghqrtk (mgr.14162) 227 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:09.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:09.310+0000 7fdc00d44700 1 -- 172.21.15.88:0/3127342966 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdbfc0f5410 msgr2=0x7fdbfc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:09.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:09.310+0000 7fdc00d44700 1 --2- 172.21.15.88:0/3127342966 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdbfc0f5410 0x7fdbfc0f5830 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7fdbf0009fb0 tx=0x7fdbf0009b30).stop 2024-02-19T22:17:09.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:09.310+0000 7fdc00d44700 1 -- 172.21.15.88:0/3127342966 shutdown_connections 2024-02-19T22:17:09.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:09.310+0000 7fdc00d44700 1 --2- 172.21.15.88:0/3127342966 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdbfc0f5410 0x7fdbfc0f5830 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:09.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:09.310+0000 7fdc00d44700 1 -- 172.21.15.88:0/3127342966 >> 172.21.15.88:0/3127342966 conn(0x7fdbfc0f0b70 msgr2=0x7fdbfc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:09.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:09.310+0000 7fdc00d44700 1 -- 172.21.15.88:0/3127342966 shutdown_connections 2024-02-19T22:17:09.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:09.310+0000 7fdc00d44700 1 -- 172.21.15.88:0/3127342966 wait complete. 2024-02-19T22:17:09.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:09.310+0000 7fdc00d44700 1 Processor -- start 2024-02-19T22:17:09.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:09.311+0000 7fdc00d44700 1 -- start start 2024-02-19T22:17:09.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:09.311+0000 7fdc00d44700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdbfc0f5410 0x7fdbfc1094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:09.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:09.311+0000 7fdc00d44700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbfc060240 con 0x7fdbfc0f5410 2024-02-19T22:17:09.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:09.311+0000 7fdbfa59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdbfc0f5410 0x7fdbfc1094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:09.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:09.311+0000 7fdbfa59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdbfc0f5410 0x7fdbfc1094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:35760/0 (socket says 172.21.15.88:35760) 2024-02-19T22:17:09.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:09.311+0000 7fdbfa59c700 1 -- 172.21.15.88:0/749720233 learned_addr learned my addr 172.21.15.88:0/749720233 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:17:09.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:09.311+0000 7fdbfa59c700 1 -- 172.21.15.88:0/749720233 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbf0005040 con 0x7fdbfc0f5410 2024-02-19T22:17:09.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:09.311+0000 7fdbfa59c700 1 --2- 172.21.15.88:0/749720233 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdbfc0f5410 0x7fdbfc1094e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7fdbf0005010 tx=0x7fdbf000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:09.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:09.311+0000 7fdbeaffd700 1 -- 172.21.15.88:0/749720233 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbf0009320 con 0x7fdbfc0f5410 2024-02-19T22:17:09.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:09.312+0000 7fdc00d44700 1 -- 172.21.15.88:0/749720233 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbfc060440 con 0x7fdbfc0f5410 2024-02-19T22:17:09.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:09.312+0000 7fdbeaffd700 1 -- 172.21.15.88:0/749720233 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdbf0013380 con 0x7fdbfc0f5410 2024-02-19T22:17:09.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:09.312+0000 7fdbeaffd700 1 -- 172.21.15.88:0/749720233 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbf0014e50 con 0x7fdbfc0f5410 2024-02-19T22:17:09.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:09.312+0000 7fdc00d44700 1 -- 172.21.15.88:0/749720233 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbfc109b90 con 0x7fdbfc0f5410 2024-02-19T22:17:09.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:09.312+0000 7fdbeaffd700 1 -- 172.21.15.88:0/749720233 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdbf00095f0 con 0x7fdbfc0f5410 2024-02-19T22:17:09.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:09.312+0000 7fdc00d44700 1 -- 172.21.15.88:0/749720233 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbfc045ad0 con 0x7fdbfc0f5410 2024-02-19T22:17:09.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:09.312+0000 7fdbeaffd700 1 --2- 172.21.15.88:0/749720233 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdbe402ff10 0x7fdbe40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:09.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:09.313+0000 7fdbeaffd700 1 -- 172.21.15.88:0/749720233 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdbf0040e90 con 0x7fdbfc0f5410 2024-02-19T22:17:09.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:09.313+0000 7fdbf9d9b700 1 --2- 172.21.15.88:0/749720233 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdbe402ff10 0x7fdbe40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:09.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:09.313+0000 7fdbf9d9b700 1 --2- 172.21.15.88:0/749720233 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdbe402ff10 0x7fdbe40323d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fdbec0096a0 tx=0x7fdbec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:09.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:09.317+0000 7fdbeaffd700 1 -- 172.21.15.88:0/749720233 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdbf0014420 con 0x7fdbfc0f5410 2024-02-19T22:17:09.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:09.521+0000 7fdc00d44700 1 -- 172.21.15.88:0/749720233 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdbfc10a3b0 con 0x7fdbfc0f5410 2024-02-19T22:17:09.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:09.521+0000 7fdbeaffd700 1 -- 172.21.15.88:0/749720233 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdbf0010230 con 0x7fdbfc0f5410 2024-02-19T22:17:09.523 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:17:09.523 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:09.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:09.523+0000 7fdc00d44700 1 -- 172.21.15.88:0/749720233 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdbe402ff10 msgr2=0x7fdbe40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:09.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:09.523+0000 7fdc00d44700 1 --2- 172.21.15.88:0/749720233 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdbe402ff10 0x7fdbe40323d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fdbec0096a0 tx=0x7fdbec006b40).stop 2024-02-19T22:17:09.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:09.523+0000 7fdc00d44700 1 -- 172.21.15.88:0/749720233 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdbfc0f5410 msgr2=0x7fdbfc1094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:09.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:09.523+0000 7fdc00d44700 1 --2- 172.21.15.88:0/749720233 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdbfc0f5410 0x7fdbfc1094e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7fdbf0005010 tx=0x7fdbf000b040).stop 2024-02-19T22:17:09.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:09.523+0000 7fdc00d44700 1 -- 172.21.15.88:0/749720233 shutdown_connections 2024-02-19T22:17:09.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:09.523+0000 7fdc00d44700 1 --2- 172.21.15.88:0/749720233 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdbe402ff10 0x7fdbe40323d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:09.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:09.523+0000 7fdc00d44700 1 --2- 172.21.15.88:0/749720233 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdbfc0f5410 0x7fdbfc1094e0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:09.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:09.523+0000 7fdc00d44700 1 -- 172.21.15.88:0/749720233 >> 172.21.15.88:0/749720233 conn(0x7fdbfc0f0b70 msgr2=0x7fdbfc0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:09.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:09.523+0000 7fdc00d44700 1 -- 172.21.15.88:0/749720233 shutdown_connections 2024-02-19T22:17:09.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:09.523+0000 7fdc00d44700 1 -- 172.21.15.88:0/749720233 wait complete. 2024-02-19T22:17:09.525 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:17:09.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:09 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:17:07.687475+0000 mgr.smithi052.ghqrtk (mgr.14162) 228 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:10.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:10 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:17:09.521683+0000 mon.smithi052 (mon.0) 250 : audit [DBG] from='client.? 172.21.15.88:0/749720233' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:11.230 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:11.230 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:11.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:11 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:17:09.687771+0000 mgr.smithi052.ghqrtk (mgr.14162) 229 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:13.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:13.057+0000 7f82589b7700 1 -- 172.21.15.88:0/4136381640 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82540f6510 msgr2=0x7f82540f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:13.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:13.057+0000 7f82589b7700 1 --2- 172.21.15.88:0/4136381640 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82540f6510 0x7f82540f6930 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f8248009fb0 tx=0x7f8248009b30).stop 2024-02-19T22:17:13.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:13.057+0000 7f82589b7700 1 -- 172.21.15.88:0/4136381640 shutdown_connections 2024-02-19T22:17:13.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:13.057+0000 7f82589b7700 1 --2- 172.21.15.88:0/4136381640 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82540f6510 0x7f82540f6930 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:13.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:13.057+0000 7f82589b7700 1 -- 172.21.15.88:0/4136381640 >> 172.21.15.88:0/4136381640 conn(0x7f82540f1c70 msgr2=0x7f82540f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:13.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:13.057+0000 7f82589b7700 1 -- 172.21.15.88:0/4136381640 shutdown_connections 2024-02-19T22:17:13.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:13.057+0000 7f82589b7700 1 -- 172.21.15.88:0/4136381640 wait complete. 2024-02-19T22:17:13.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:13.058+0000 7f82589b7700 1 Processor -- start 2024-02-19T22:17:13.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:13.058+0000 7f82589b7700 1 -- start start 2024-02-19T22:17:13.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:13.058+0000 7f82589b7700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82540f6510 0x7f82541067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:13.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:13.058+0000 7f82589b7700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8254106cd0 con 0x7f82540f6510 2024-02-19T22:17:13.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:13.058+0000 7f825259c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82540f6510 0x7f82541067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:13.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:13.058+0000 7f825259c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82540f6510 0x7f82541067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:35780/0 (socket says 172.21.15.88:35780) 2024-02-19T22:17:13.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:13.058+0000 7f825259c700 1 -- 172.21.15.88:0/601699177 learned_addr learned my addr 172.21.15.88:0/601699177 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:17:13.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:13.058+0000 7f825259c700 1 -- 172.21.15.88:0/601699177 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8248005040 con 0x7f82540f6510 2024-02-19T22:17:13.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:13.058+0000 7f825259c700 1 --2- 172.21.15.88:0/601699177 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82540f6510 0x7f82541067e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f8248005010 tx=0x7f824800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:13.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:13.058+0000 7f8242ffd700 1 -- 172.21.15.88:0/601699177 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8248009320 con 0x7f82540f6510 2024-02-19T22:17:13.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:13.059+0000 7f82589b7700 1 -- 172.21.15.88:0/601699177 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8254106ed0 con 0x7f82540f6510 2024-02-19T22:17:13.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:13.059+0000 7f8242ffd700 1 -- 172.21.15.88:0/601699177 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8248013380 con 0x7f82540f6510 2024-02-19T22:17:13.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:13.059+0000 7f8242ffd700 1 -- 172.21.15.88:0/601699177 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8248014e50 con 0x7f82540f6510 2024-02-19T22:17:13.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:13.059+0000 7f82589b7700 1 -- 172.21.15.88:0/601699177 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8254107360 con 0x7f82540f6510 2024-02-19T22:17:13.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:13.059+0000 7f8242ffd700 1 -- 172.21.15.88:0/601699177 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8248014420 con 0x7f82540f6510 2024-02-19T22:17:13.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:13.059+0000 7f82589b7700 1 -- 172.21.15.88:0/601699177 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8254045ad0 con 0x7f82540f6510 2024-02-19T22:17:13.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:13.059+0000 7f8242ffd700 1 --2- 172.21.15.88:0/601699177 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f823c02ff10 0x7f823c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:13.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:13.060+0000 7f8242ffd700 1 -- 172.21.15.88:0/601699177 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8248040e30 con 0x7f82540f6510 2024-02-19T22:17:13.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:13.060+0000 7f8251d9b700 1 --2- 172.21.15.88:0/601699177 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f823c02ff10 0x7f823c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:13.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:13.060+0000 7f8251d9b700 1 --2- 172.21.15.88:0/601699177 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f823c02ff10 0x7f823c0323d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f82440096a0 tx=0x7f8244006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:13.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:13.064+0000 7f8242ffd700 1 -- 172.21.15.88:0/601699177 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f82480095f0 con 0x7f82540f6510 2024-02-19T22:17:13.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:13.268+0000 7f82589b7700 1 -- 172.21.15.88:0/601699177 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8254108210 con 0x7f82540f6510 2024-02-19T22:17:13.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:13.269+0000 7f8242ffd700 1 -- 172.21.15.88:0/601699177 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f824801db70 con 0x7f82540f6510 2024-02-19T22:17:13.270 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:17:13.270 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:13.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:13.270+0000 7f82589b7700 1 -- 172.21.15.88:0/601699177 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f823c02ff10 msgr2=0x7f823c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:13.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:13.270+0000 7f82589b7700 1 --2- 172.21.15.88:0/601699177 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f823c02ff10 0x7f823c0323d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f82440096a0 tx=0x7f8244006b40).stop 2024-02-19T22:17:13.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:13.270+0000 7f82589b7700 1 -- 172.21.15.88:0/601699177 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82540f6510 msgr2=0x7f82541067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:13.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:13.270+0000 7f82589b7700 1 --2- 172.21.15.88:0/601699177 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82540f6510 0x7f82541067e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f8248005010 tx=0x7f824800b040).stop 2024-02-19T22:17:13.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:13.270+0000 7f82589b7700 1 -- 172.21.15.88:0/601699177 shutdown_connections 2024-02-19T22:17:13.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:13.270+0000 7f82589b7700 1 --2- 172.21.15.88:0/601699177 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f823c02ff10 0x7f823c0323d0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:13.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:13.270+0000 7f82589b7700 1 --2- 172.21.15.88:0/601699177 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82540f6510 0x7f82541067e0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:13.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:13.270+0000 7f82589b7700 1 -- 172.21.15.88:0/601699177 >> 172.21.15.88:0/601699177 conn(0x7f82540f1c70 msgr2=0x7f82540f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:13.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:13.270+0000 7f82589b7700 1 -- 172.21.15.88:0/601699177 shutdown_connections 2024-02-19T22:17:13.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:13.270+0000 7f82589b7700 1 -- 172.21.15.88:0/601699177 wait complete. 2024-02-19T22:17:13.273 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:17:13.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:13 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:17:11.688117+0000 mgr.smithi052.ghqrtk (mgr.14162) 230 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:14.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:14 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:17:13.269187+0000 mon.smithi052 (mon.0) 251 : audit [DBG] from='client.? 172.21.15.88:0/601699177' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:14.991 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:14.991 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:15.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:15 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:17:13.688449+0000 mgr.smithi052.ghqrtk (mgr.14162) 231 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:16.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:16.888+0000 7fdb3153a700 1 -- 172.21.15.88:0/1093606206 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdb2c0f6510 msgr2=0x7fdb2c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:16.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:16.888+0000 7fdb3153a700 1 --2- 172.21.15.88:0/1093606206 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdb2c0f6510 0x7fdb2c0f6930 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7fdb1c009fb0 tx=0x7fdb1c009b30).stop 2024-02-19T22:17:16.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:16.888+0000 7fdb3153a700 1 -- 172.21.15.88:0/1093606206 shutdown_connections 2024-02-19T22:17:16.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:16.888+0000 7fdb3153a700 1 --2- 172.21.15.88:0/1093606206 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdb2c0f6510 0x7fdb2c0f6930 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:16.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:16.888+0000 7fdb3153a700 1 -- 172.21.15.88:0/1093606206 >> 172.21.15.88:0/1093606206 conn(0x7fdb2c0f1c70 msgr2=0x7fdb2c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:16.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:16.888+0000 7fdb3153a700 1 -- 172.21.15.88:0/1093606206 shutdown_connections 2024-02-19T22:17:16.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:16.888+0000 7fdb3153a700 1 -- 172.21.15.88:0/1093606206 wait complete. 2024-02-19T22:17:16.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:16.889+0000 7fdb3153a700 1 Processor -- start 2024-02-19T22:17:16.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:16.889+0000 7fdb3153a700 1 -- start start 2024-02-19T22:17:16.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:16.889+0000 7fdb3153a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdb2c0f6510 0x7fdb2c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:16.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:16.889+0000 7fdb3153a700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2c106cd0 con 0x7fdb2c0f6510 2024-02-19T22:17:16.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:16.889+0000 7fdb2affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdb2c0f6510 0x7fdb2c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:16.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:16.889+0000 7fdb2affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdb2c0f6510 0x7fdb2c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:53270/0 (socket says 172.21.15.88:53270) 2024-02-19T22:17:16.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:16.889+0000 7fdb2affd700 1 -- 172.21.15.88:0/1752880727 learned_addr learned my addr 172.21.15.88:0/1752880727 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:17:16.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:16.890+0000 7fdb2affd700 1 -- 172.21.15.88:0/1752880727 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb1c005040 con 0x7fdb2c0f6510 2024-02-19T22:17:16.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:16.890+0000 7fdb2affd700 1 --2- 172.21.15.88:0/1752880727 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdb2c0f6510 0x7fdb2c1067e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7fdb1c005010 tx=0x7fdb1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:16.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:16.890+0000 7fdb137fe700 1 -- 172.21.15.88:0/1752880727 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb1c009320 con 0x7fdb2c0f6510 2024-02-19T22:17:16.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:16.890+0000 7fdb137fe700 1 -- 172.21.15.88:0/1752880727 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdb1c013380 con 0x7fdb2c0f6510 2024-02-19T22:17:16.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:16.890+0000 7fdb3153a700 1 -- 172.21.15.88:0/1752880727 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb2c106ed0 con 0x7fdb2c0f6510 2024-02-19T22:17:16.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:16.890+0000 7fdb137fe700 1 -- 172.21.15.88:0/1752880727 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb1c014e50 con 0x7fdb2c0f6510 2024-02-19T22:17:16.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:16.890+0000 7fdb3153a700 1 -- 172.21.15.88:0/1752880727 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb2c107360 con 0x7fdb2c0f6510 2024-02-19T22:17:16.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:16.891+0000 7fdb137fe700 1 -- 172.21.15.88:0/1752880727 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdb1c0263a0 con 0x7fdb2c0f6510 2024-02-19T22:17:16.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:16.891+0000 7fdb137fe700 1 --2- 172.21.15.88:0/1752880727 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdb1402ff10 0x7fdb140323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:16.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:16.891+0000 7fdb137fe700 1 -- 172.21.15.88:0/1752880727 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdb1c040f90 con 0x7fdb2c0f6510 2024-02-19T22:17:16.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:16.891+0000 7fdb3153a700 1 -- 172.21.15.88:0/1752880727 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb2c045ad0 con 0x7fdb2c0f6510 2024-02-19T22:17:16.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:16.891+0000 7fdb2a7fc700 1 --2- 172.21.15.88:0/1752880727 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdb1402ff10 0x7fdb140323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:16.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:16.892+0000 7fdb2a7fc700 1 --2- 172.21.15.88:0/1752880727 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdb1402ff10 0x7fdb140323d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fdb200096a0 tx=0x7fdb20006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:16.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:16.896+0000 7fdb137fe700 1 -- 172.21.15.88:0/1752880727 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdb1c012920 con 0x7fdb2c0f6510 2024-02-19T22:17:17.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:17.106+0000 7fdb3153a700 1 -- 172.21.15.88:0/1752880727 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdb2c1080b0 con 0x7fdb2c0f6510 2024-02-19T22:17:17.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:17.106+0000 7fdb137fe700 1 -- 172.21.15.88:0/1752880727 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdb1c02b020 con 0x7fdb2c0f6510 2024-02-19T22:17:17.109 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:17:17.109 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:17.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:17.108+0000 7fdb3153a700 1 -- 172.21.15.88:0/1752880727 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdb1402ff10 msgr2=0x7fdb140323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:17.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:17.108+0000 7fdb3153a700 1 --2- 172.21.15.88:0/1752880727 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdb1402ff10 0x7fdb140323d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fdb200096a0 tx=0x7fdb20006b40).stop 2024-02-19T22:17:17.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:17.108+0000 7fdb3153a700 1 -- 172.21.15.88:0/1752880727 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdb2c0f6510 msgr2=0x7fdb2c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:17.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:17.108+0000 7fdb3153a700 1 --2- 172.21.15.88:0/1752880727 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdb2c0f6510 0x7fdb2c1067e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7fdb1c005010 tx=0x7fdb1c00b040).stop 2024-02-19T22:17:17.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:17.108+0000 7fdb3153a700 1 -- 172.21.15.88:0/1752880727 shutdown_connections 2024-02-19T22:17:17.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:17.108+0000 7fdb3153a700 1 --2- 172.21.15.88:0/1752880727 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdb1402ff10 0x7fdb140323d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:17.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:17.108+0000 7fdb3153a700 1 --2- 172.21.15.88:0/1752880727 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdb2c0f6510 0x7fdb2c1067e0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:17.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:17.108+0000 7fdb3153a700 1 -- 172.21.15.88:0/1752880727 >> 172.21.15.88:0/1752880727 conn(0x7fdb2c0f1c70 msgr2=0x7fdb2c0f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:17.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:17.108+0000 7fdb3153a700 1 -- 172.21.15.88:0/1752880727 shutdown_connections 2024-02-19T22:17:17.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:17.108+0000 7fdb3153a700 1 -- 172.21.15.88:0/1752880727 wait complete. 2024-02-19T22:17:17.111 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:17:17.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:17 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:17:15.688765+0000 mgr.smithi052.ghqrtk (mgr.14162) 232 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:17.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:17 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:17:17.107864+0000 mon.smithi052 (mon.0) 252 : audit [DBG] from='client.? 172.21.15.88:0/1752880727' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:18.595 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cephadm 2024-02-19T22:17:16.879769+0000 mgr.smithi052.ghqrtk (mgr.14162) 233 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi052 ... 2024-02-19T22:17:18.595 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: Verifying port 9100 ... 2024-02-19T22:17:18.595 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: Non-zero exit code 1 from systemctl start ceph-56739a54-cf73-11ee-95bb-87774f69a715@node-exporter.smithi052 2024-02-19T22:17:18.595 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: systemctl: stderr Job for ceph-56739a54-cf73-11ee-95bb-87774f69a715@node-exporter.smithi052.service failed because a timeout was exceeded. 2024-02-19T22:17:18.596 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: systemctl: stderr See "systemctl status ceph-56739a54-cf73-11ee-95bb-87774f69a715@node-exporter.smithi052.service" and "journalctl -xe" for details. 2024-02-19T22:17:18.596 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: Traceback (most recent call last): 2024-02-19T22:17:18.596 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: File "/var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-19T22:17:18.596 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: main() 2024-02-19T22:17:18.596 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: File "/var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-19T22:17:18.596 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: r = ctx.func(ctx) 2024-02-19T22:17:18.596 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: File "/var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-19T22:17:18.596 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: return func(ctx) 2024-02-19T22:17:18.596 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: File "/var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-02-19T22:17:18.596 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: ports=daemon_ports) 2024-02-19T22:17:18.596 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: File "/var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-02-19T22:17:18.596 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: c, osd_fsid=osd_fsid, ports=ports) 2024-02-19T22:17:18.596 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: File "/var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-02-19T22:17:18.597 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-02-19T22:17:18.597 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: File "/var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-02-19T22:17:18.597 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-02-19T22:17:18.597 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: RuntimeError: Failed command: systemctl start ceph-56739a54-cf73-11ee-95bb-87774f69a715@node-exporter.smithi052 2024-02-19T22:17:18.597 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: Traceback (most recent call last): 2024-02-19T22:17:18.597 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-02-19T22:17:18.597 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: yield (conn, connr) 2024-02-19T22:17:18.597 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-02-19T22:17:18.597 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: code, '\n'.join(err))) 2024-02-19T22:17:18.597 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi052 ... 2024-02-19T22:17:18.597 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: Verifying port 9100 ... 2024-02-19T22:17:18.597 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: Non-zero exit code 1 from systemctl start ceph-56739a54-cf73-11ee-95bb-87774f69a715@node-exporter.smithi052 2024-02-19T22:17:18.597 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: systemctl: stderr Job for ceph-56739a54-cf73-11ee-95bb-87774f69a715@node-exporter.smithi052.service failed because a timeout was exceeded. 2024-02-19T22:17:18.597 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: systemctl: stderr See "systemctl status ceph-56739a54-cf73-11ee-95bb-87774f69a715@node-exporter.smithi052.service" and "journalctl -xe" for details. 2024-02-19T22:17:18.597 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: Traceback (most recent call last): 2024-02-19T22:17:18.598 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: File "/var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-19T22:17:18.598 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: main() 2024-02-19T22:17:18.598 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: File "/var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-19T22:17:18.598 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: r = ctx.func(ctx) 2024-02-19T22:17:18.598 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: File "/var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-19T22:17:18.598 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: return func(ctx) 2024-02-19T22:17:18.598 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: File "/var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-02-19T22:17:18.598 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: ports=daemon_ports) 2024-02-19T22:17:18.598 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: File "/var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-02-19T22:17:18.598 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: c, osd_fsid=osd_fsid, ports=ports) 2024-02-19T22:17:18.598 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: File "/var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-02-19T22:17:18.598 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-02-19T22:17:18.598 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: File "/var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-02-19T22:17:18.598 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-02-19T22:17:18.598 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: RuntimeError: Failed command: systemctl start ceph-56739a54-cf73-11ee-95bb-87774f69a715@node-exporter.smithi052 2024-02-19T22:17:18.599 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cephadm 2024-02-19T22:17:16.885347+0000 mgr.smithi052.ghqrtk (mgr.14162) 234 : cephadm [INF] Deploying daemon prometheus.smithi052 on smithi052 2024-02-19T22:17:18.688 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:18.688 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:19.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:19 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:17:17.689068+0000 mgr.smithi052.ghqrtk (mgr.14162) 235 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:20.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:20.619+0000 7f980d220700 1 -- 172.21.15.88:0/2437919511 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f98080f62a0 msgr2=0x7f98080f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:20.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:20.619+0000 7f980d220700 1 --2- 172.21.15.88:0/2437919511 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f98080f62a0 0x7f98080f66c0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f97f8009fb0 tx=0x7f97f8009b30).stop 2024-02-19T22:17:20.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:20.619+0000 7f980d220700 1 -- 172.21.15.88:0/2437919511 shutdown_connections 2024-02-19T22:17:20.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:20.619+0000 7f980d220700 1 --2- 172.21.15.88:0/2437919511 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f98080f62a0 0x7f98080f66c0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:20.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:20.619+0000 7f980d220700 1 -- 172.21.15.88:0/2437919511 >> 172.21.15.88:0/2437919511 conn(0x7f98080f1aa0 msgr2=0x7f98080f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:20.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:20.619+0000 7f980d220700 1 -- 172.21.15.88:0/2437919511 shutdown_connections 2024-02-19T22:17:20.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:20.619+0000 7f980d220700 1 -- 172.21.15.88:0/2437919511 wait complete. 2024-02-19T22:17:20.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:20.619+0000 7f980d220700 1 Processor -- start 2024-02-19T22:17:20.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:20.619+0000 7f980d220700 1 -- start start 2024-02-19T22:17:20.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:20.619+0000 7f980d220700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f98080f62a0 0x7f98081065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:20.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:20.620+0000 7f980d220700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9808106ac0 con 0x7f98080f62a0 2024-02-19T22:17:20.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:20.620+0000 7f9807fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f98080f62a0 0x7f98081065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:20.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:20.620+0000 7f9807fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f98080f62a0 0x7f98081065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:53300/0 (socket says 172.21.15.88:53300) 2024-02-19T22:17:20.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:20.620+0000 7f9807fff700 1 -- 172.21.15.88:0/2803487502 learned_addr learned my addr 172.21.15.88:0/2803487502 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:17:20.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:20.620+0000 7f9807fff700 1 -- 172.21.15.88:0/2803487502 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97f8005040 con 0x7f98080f62a0 2024-02-19T22:17:20.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:20.620+0000 7f9807fff700 1 --2- 172.21.15.88:0/2803487502 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f98080f62a0 0x7f98081065d0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f97f80089f0 tx=0x7f97f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:20.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:20.620+0000 7f9804ff9700 1 -- 172.21.15.88:0/2803487502 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f97f8009320 con 0x7f98080f62a0 2024-02-19T22:17:20.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:20.620+0000 7f980d220700 1 -- 172.21.15.88:0/2803487502 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9808106cc0 con 0x7f98080f62a0 2024-02-19T22:17:20.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:20.620+0000 7f9804ff9700 1 -- 172.21.15.88:0/2803487502 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f97f8013380 con 0x7f98080f62a0 2024-02-19T22:17:20.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:20.620+0000 7f9804ff9700 1 -- 172.21.15.88:0/2803487502 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f97f8014e20 con 0x7f98080f62a0 2024-02-19T22:17:20.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:20.621+0000 7f980d220700 1 -- 172.21.15.88:0/2803487502 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9808107150 con 0x7f98080f62a0 2024-02-19T22:17:20.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:20.621+0000 7f9804ff9700 1 -- 172.21.15.88:0/2803487502 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f97f80263a0 con 0x7f98080f62a0 2024-02-19T22:17:20.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:20.621+0000 7f9804ff9700 1 --2- 172.21.15.88:0/2803487502 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f97f002ff10 0x7f97f00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:20.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:20.621+0000 7f980d220700 1 -- 172.21.15.88:0/2803487502 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9808045ad0 con 0x7f98080f62a0 2024-02-19T22:17:20.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:20.622+0000 7f9804ff9700 1 -- 172.21.15.88:0/2803487502 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f97f8040fa0 con 0x7f98080f62a0 2024-02-19T22:17:20.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:20.622+0000 7f98077fe700 1 --2- 172.21.15.88:0/2803487502 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f97f002ff10 0x7f97f00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:20.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:20.622+0000 7f98077fe700 1 --2- 172.21.15.88:0/2803487502 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f97f002ff10 0x7f97f00323d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f97fc0096a0 tx=0x7f97fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:20.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:20.626+0000 7f9804ff9700 1 -- 172.21.15.88:0/2803487502 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f97f8012920 con 0x7f98080f62a0 2024-02-19T22:17:20.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:20.830+0000 7f980d220700 1 -- 172.21.15.88:0/2803487502 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9808108000 con 0x7f98080f62a0 2024-02-19T22:17:20.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:20.831+0000 7f9804ff9700 1 -- 172.21.15.88:0/2803487502 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f97f80126f0 con 0x7f98080f62a0 2024-02-19T22:17:20.833 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:17:20.833 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:20.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:20.832+0000 7f980d220700 1 -- 172.21.15.88:0/2803487502 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f97f002ff10 msgr2=0x7f97f00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:20.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:20.832+0000 7f980d220700 1 --2- 172.21.15.88:0/2803487502 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f97f002ff10 0x7f97f00323d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f97fc0096a0 tx=0x7f97fc006b40).stop 2024-02-19T22:17:20.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:20.833+0000 7f980d220700 1 -- 172.21.15.88:0/2803487502 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f98080f62a0 msgr2=0x7f98081065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:20.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:20.833+0000 7f980d220700 1 --2- 172.21.15.88:0/2803487502 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f98080f62a0 0x7f98081065d0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f97f80089f0 tx=0x7f97f800b040).stop 2024-02-19T22:17:20.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:20.833+0000 7f980d220700 1 -- 172.21.15.88:0/2803487502 shutdown_connections 2024-02-19T22:17:20.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:20.833+0000 7f980d220700 1 --2- 172.21.15.88:0/2803487502 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f97f002ff10 0x7f97f00323d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:20.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:20.833+0000 7f980d220700 1 --2- 172.21.15.88:0/2803487502 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f98080f62a0 0x7f98081065d0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:20.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:20.833+0000 7f980d220700 1 -- 172.21.15.88:0/2803487502 >> 172.21.15.88:0/2803487502 conn(0x7f98080f1aa0 msgr2=0x7f98080f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:20.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:20.833+0000 7f980d220700 1 -- 172.21.15.88:0/2803487502 shutdown_connections 2024-02-19T22:17:20.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:20.833+0000 7f980d220700 1 -- 172.21.15.88:0/2803487502 wait complete. 2024-02-19T22:17:20.836 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:17:21.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:21 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:17:19.689344+0000 mgr.smithi052.ghqrtk (mgr.14162) 236 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:21.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:21 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:17:20.832502+0000 mon.smithi052 (mon.0) 253 : audit [DBG] from='client.? 172.21.15.88:0/2803487502' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:22.548 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:22.549 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:23.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:23 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:17:21.689614+0000 mgr.smithi052.ghqrtk (mgr.14162) 237 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:24.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:24.581+0000 7fbdfa340700 1 -- 172.21.15.88:0/1402620617 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbdf40f5410 msgr2=0x7fbdf40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:24.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:24.581+0000 7fbdfa340700 1 --2- 172.21.15.88:0/1402620617 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbdf40f5410 0x7fbdf40f5830 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7fbde4009fb0 tx=0x7fbde4009b30).stop 2024-02-19T22:17:24.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:24.581+0000 7fbdfa340700 1 -- 172.21.15.88:0/1402620617 shutdown_connections 2024-02-19T22:17:24.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:24.581+0000 7fbdfa340700 1 --2- 172.21.15.88:0/1402620617 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbdf40f5410 0x7fbdf40f5830 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:24.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:24.581+0000 7fbdfa340700 1 -- 172.21.15.88:0/1402620617 >> 172.21.15.88:0/1402620617 conn(0x7fbdf40f0b70 msgr2=0x7fbdf40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:24.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:24.582+0000 7fbdfa340700 1 -- 172.21.15.88:0/1402620617 shutdown_connections 2024-02-19T22:17:24.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:24.582+0000 7fbdfa340700 1 -- 172.21.15.88:0/1402620617 wait complete. 2024-02-19T22:17:24.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:24.582+0000 7fbdfa340700 1 Processor -- start 2024-02-19T22:17:24.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:24.582+0000 7fbdfa340700 1 -- start start 2024-02-19T22:17:24.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:24.582+0000 7fbdfa340700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbdf40f5410 0x7fbdf41067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:24.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:24.582+0000 7fbdfa340700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf4106cd0 con 0x7fbdf40f5410 2024-02-19T22:17:24.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:24.583+0000 7fbdf3fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbdf40f5410 0x7fbdf41067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:24.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:24.583+0000 7fbdf3fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbdf40f5410 0x7fbdf41067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:56512/0 (socket says 172.21.15.88:56512) 2024-02-19T22:17:24.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:24.583+0000 7fbdf3fff700 1 -- 172.21.15.88:0/4062183228 learned_addr learned my addr 172.21.15.88:0/4062183228 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:17:24.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:24.583+0000 7fbdf3fff700 1 -- 172.21.15.88:0/4062183228 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde4005040 con 0x7fbdf40f5410 2024-02-19T22:17:24.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:24.583+0000 7fbdf3fff700 1 --2- 172.21.15.88:0/4062183228 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbdf40f5410 0x7fbdf41067e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7fbde4005010 tx=0x7fbde400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:24.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:24.583+0000 7fbdf0ff9700 1 -- 172.21.15.88:0/4062183228 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbde4009320 con 0x7fbdf40f5410 2024-02-19T22:17:24.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:24.583+0000 7fbdfa340700 1 -- 172.21.15.88:0/4062183228 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdf4106ed0 con 0x7fbdf40f5410 2024-02-19T22:17:24.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:24.583+0000 7fbdf0ff9700 1 -- 172.21.15.88:0/4062183228 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbde4013380 con 0x7fbdf40f5410 2024-02-19T22:17:24.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:24.583+0000 7fbdf0ff9700 1 -- 172.21.15.88:0/4062183228 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbde4014e50 con 0x7fbdf40f5410 2024-02-19T22:17:24.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:24.583+0000 7fbdfa340700 1 -- 172.21.15.88:0/4062183228 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdf4107360 con 0x7fbdf40f5410 2024-02-19T22:17:24.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:24.584+0000 7fbdf0ff9700 1 -- 172.21.15.88:0/4062183228 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbde4014420 con 0x7fbdf40f5410 2024-02-19T22:17:24.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:24.584+0000 7fbdfa340700 1 -- 172.21.15.88:0/4062183228 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdf4045ad0 con 0x7fbdf40f5410 2024-02-19T22:17:24.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:24.584+0000 7fbdf0ff9700 1 --2- 172.21.15.88:0/4062183228 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fbddc02ff10 0x7fbddc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:24.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:24.584+0000 7fbdf0ff9700 1 -- 172.21.15.88:0/4062183228 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbde4040eb0 con 0x7fbdf40f5410 2024-02-19T22:17:24.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:24.584+0000 7fbdf37fe700 1 --2- 172.21.15.88:0/4062183228 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fbddc02ff10 0x7fbddc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:24.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:24.585+0000 7fbdf37fe700 1 --2- 172.21.15.88:0/4062183228 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fbddc02ff10 0x7fbddc0323d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fbde80096a0 tx=0x7fbde8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:24.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:24.589+0000 7fbdf0ff9700 1 -- 172.21.15.88:0/4062183228 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbde40095f0 con 0x7fbdf40f5410 2024-02-19T22:17:24.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:24.798+0000 7fbdfa340700 1 -- 172.21.15.88:0/4062183228 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbdf4108210 con 0x7fbdf40f5410 2024-02-19T22:17:24.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:24.798+0000 7fbdf0ff9700 1 -- 172.21.15.88:0/4062183228 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbde401db70 con 0x7fbdf40f5410 2024-02-19T22:17:24.801 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:17:24.801 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:24.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:24.800+0000 7fbdfa340700 1 -- 172.21.15.88:0/4062183228 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fbddc02ff10 msgr2=0x7fbddc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:24.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:24.800+0000 7fbdfa340700 1 --2- 172.21.15.88:0/4062183228 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fbddc02ff10 0x7fbddc0323d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fbde80096a0 tx=0x7fbde8006b40).stop 2024-02-19T22:17:24.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:24.800+0000 7fbdfa340700 1 -- 172.21.15.88:0/4062183228 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbdf40f5410 msgr2=0x7fbdf41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:24.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:24.800+0000 7fbdfa340700 1 --2- 172.21.15.88:0/4062183228 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbdf40f5410 0x7fbdf41067e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7fbde4005010 tx=0x7fbde400b040).stop 2024-02-19T22:17:24.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:24.800+0000 7fbdfa340700 1 -- 172.21.15.88:0/4062183228 shutdown_connections 2024-02-19T22:17:24.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:24.800+0000 7fbdfa340700 1 --2- 172.21.15.88:0/4062183228 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fbddc02ff10 0x7fbddc0323d0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:24.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:24.800+0000 7fbdfa340700 1 --2- 172.21.15.88:0/4062183228 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbdf40f5410 0x7fbdf41067e0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:24.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:24.800+0000 7fbdfa340700 1 -- 172.21.15.88:0/4062183228 >> 172.21.15.88:0/4062183228 conn(0x7fbdf40f0b70 msgr2=0x7fbdf40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:24.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:24.800+0000 7fbdfa340700 1 -- 172.21.15.88:0/4062183228 shutdown_connections 2024-02-19T22:17:24.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:24.800+0000 7fbdfa340700 1 -- 172.21.15.88:0/4062183228 wait complete. 2024-02-19T22:17:24.803 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:17:25.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:25 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:17:23.689958+0000 mgr.smithi052.ghqrtk (mgr.14162) 238 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:25.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:25 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:17:24.799764+0000 mon.smithi052 (mon.0) 254 : audit [DBG] from='client.? 172.21.15.88:0/4062183228' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:26.458 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:26.458 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:27.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:27 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:17:25.690320+0000 mgr.smithi052.ghqrtk (mgr.14162) 239 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:28.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:28.307+0000 7f0d7a7af700 1 -- 172.21.15.88:0/2235677388 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0d740f51a0 msgr2=0x7f0d740f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:28.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:28.307+0000 7f0d7a7af700 1 --2- 172.21.15.88:0/2235677388 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0d740f51a0 0x7f0d740f55c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f0d70009fb0 tx=0x7f0d70009b30).stop 2024-02-19T22:17:28.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:28.307+0000 7f0d7a7af700 1 -- 172.21.15.88:0/2235677388 shutdown_connections 2024-02-19T22:17:28.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:28.307+0000 7f0d7a7af700 1 --2- 172.21.15.88:0/2235677388 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0d740f51a0 0x7f0d740f55c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:28.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:28.307+0000 7f0d7a7af700 1 -- 172.21.15.88:0/2235677388 >> 172.21.15.88:0/2235677388 conn(0x7f0d740f09a0 msgr2=0x7f0d740f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:28.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:28.307+0000 7f0d7a7af700 1 -- 172.21.15.88:0/2235677388 shutdown_connections 2024-02-19T22:17:28.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:28.307+0000 7f0d7a7af700 1 -- 172.21.15.88:0/2235677388 wait complete. 2024-02-19T22:17:28.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:28.307+0000 7f0d7a7af700 1 Processor -- start 2024-02-19T22:17:28.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:28.308+0000 7f0d7a7af700 1 -- start start 2024-02-19T22:17:28.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:28.308+0000 7f0d7a7af700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0d740f51a0 0x7f0d741092d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:28.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:28.308+0000 7f0d7a7af700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d74060240 con 0x7f0d740f51a0 2024-02-19T22:17:28.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:28.308+0000 7f0d797ad700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0d740f51a0 0x7f0d741092d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:28.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:28.308+0000 7f0d797ad700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0d740f51a0 0x7f0d741092d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:56524/0 (socket says 172.21.15.88:56524) 2024-02-19T22:17:28.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:28.308+0000 7f0d797ad700 1 -- 172.21.15.88:0/1181670876 learned_addr learned my addr 172.21.15.88:0/1181670876 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:17:28.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:28.308+0000 7f0d797ad700 1 -- 172.21.15.88:0/1181670876 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d70005040 con 0x7f0d740f51a0 2024-02-19T22:17:28.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:28.308+0000 7f0d797ad700 1 --2- 172.21.15.88:0/1181670876 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0d740f51a0 0x7f0d741092d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f0d700089f0 tx=0x7f0d7000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:28.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:28.308+0000 7f0d69ffb700 1 -- 172.21.15.88:0/1181670876 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d70009320 con 0x7f0d740f51a0 2024-02-19T22:17:28.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:28.308+0000 7f0d7a7af700 1 -- 172.21.15.88:0/1181670876 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d74060440 con 0x7f0d740f51a0 2024-02-19T22:17:28.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:28.309+0000 7f0d69ffb700 1 -- 172.21.15.88:0/1181670876 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0d70013380 con 0x7f0d740f51a0 2024-02-19T22:17:28.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:28.309+0000 7f0d69ffb700 1 -- 172.21.15.88:0/1181670876 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d70014e20 con 0x7f0d740f51a0 2024-02-19T22:17:28.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:28.309+0000 7f0d7a7af700 1 -- 172.21.15.88:0/1181670876 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d74109980 con 0x7f0d740f51a0 2024-02-19T22:17:28.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:28.310+0000 7f0d69ffb700 1 -- 172.21.15.88:0/1181670876 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0d700095f0 con 0x7f0d740f51a0 2024-02-19T22:17:28.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:28.310+0000 7f0d7a7af700 1 -- 172.21.15.88:0/1181670876 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d74045ad0 con 0x7f0d740f51a0 2024-02-19T22:17:28.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:28.310+0000 7f0d69ffb700 1 --2- 172.21.15.88:0/1181670876 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f0d6002ff60 0x7f0d60032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:28.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:28.310+0000 7f0d69ffb700 1 -- 172.21.15.88:0/1181670876 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0d70040e40 con 0x7f0d740f51a0 2024-02-19T22:17:28.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:28.310+0000 7f0d78fac700 1 --2- 172.21.15.88:0/1181670876 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f0d6002ff60 0x7f0d60032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:28.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:28.310+0000 7f0d78fac700 1 --2- 172.21.15.88:0/1181670876 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f0d6002ff60 0x7f0d60032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f0d640096a0 tx=0x7f0d64006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:28.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:28.314+0000 7f0d69ffb700 1 -- 172.21.15.88:0/1181670876 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0d7001d7f0 con 0x7f0d740f51a0 2024-02-19T22:17:28.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:28.520+0000 7f0d7a7af700 1 -- 172.21.15.88:0/1181670876 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0d7410a190 con 0x7f0d740f51a0 2024-02-19T22:17:28.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:28.521+0000 7f0d69ffb700 1 -- 172.21.15.88:0/1181670876 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0d70010230 con 0x7f0d740f51a0 2024-02-19T22:17:28.523 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:17:28.523 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:28.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:28.523+0000 7f0d7a7af700 1 -- 172.21.15.88:0/1181670876 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f0d6002ff60 msgr2=0x7f0d60032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:28.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:28.523+0000 7f0d7a7af700 1 --2- 172.21.15.88:0/1181670876 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f0d6002ff60 0x7f0d60032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f0d640096a0 tx=0x7f0d64006b40).stop 2024-02-19T22:17:28.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:28.523+0000 7f0d7a7af700 1 -- 172.21.15.88:0/1181670876 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0d740f51a0 msgr2=0x7f0d741092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:28.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:28.523+0000 7f0d7a7af700 1 --2- 172.21.15.88:0/1181670876 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0d740f51a0 0x7f0d741092d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f0d700089f0 tx=0x7f0d7000b040).stop 2024-02-19T22:17:28.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:28.523+0000 7f0d7a7af700 1 -- 172.21.15.88:0/1181670876 shutdown_connections 2024-02-19T22:17:28.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:28.523+0000 7f0d7a7af700 1 --2- 172.21.15.88:0/1181670876 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f0d6002ff60 0x7f0d60032420 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:28.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:28.523+0000 7f0d7a7af700 1 --2- 172.21.15.88:0/1181670876 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0d740f51a0 0x7f0d741092d0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:28.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:28.523+0000 7f0d7a7af700 1 -- 172.21.15.88:0/1181670876 >> 172.21.15.88:0/1181670876 conn(0x7f0d740f09a0 msgr2=0x7f0d740f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:28.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:28.523+0000 7f0d7a7af700 1 -- 172.21.15.88:0/1181670876 shutdown_connections 2024-02-19T22:17:28.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:28.523+0000 7f0d7a7af700 1 -- 172.21.15.88:0/1181670876 wait complete. 2024-02-19T22:17:28.526 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:17:29.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:29 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:17:27.690689+0000 mgr.smithi052.ghqrtk (mgr.14162) 240 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:29.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:29 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:17:28.522300+0000 mon.smithi052 (mon.0) 255 : audit [DBG] from='client.? 172.21.15.88:0/1181670876' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:30.230 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:30.230 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:31.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:31 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:17:29.691080+0000 mgr.smithi052.ghqrtk (mgr.14162) 241 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:32.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:32.100+0000 7fd874aec700 1 -- 172.21.15.88:0/2591153804 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8700f4740 msgr2=0x7fd8700f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:32.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:32.100+0000 7fd874aec700 1 --2- 172.21.15.88:0/2591153804 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8700f4740 0x7fd8700f4b60 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7fd864009fb0 tx=0x7fd864009b30).stop 2024-02-19T22:17:32.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:32.100+0000 7fd874aec700 1 -- 172.21.15.88:0/2591153804 shutdown_connections 2024-02-19T22:17:32.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:32.100+0000 7fd874aec700 1 --2- 172.21.15.88:0/2591153804 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8700f4740 0x7fd8700f4b60 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:32.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:32.100+0000 7fd874aec700 1 -- 172.21.15.88:0/2591153804 >> 172.21.15.88:0/2591153804 conn(0x7fd8700f0260 msgr2=0x7fd8700f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:32.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:32.100+0000 7fd874aec700 1 -- 172.21.15.88:0/2591153804 shutdown_connections 2024-02-19T22:17:32.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:32.100+0000 7fd874aec700 1 -- 172.21.15.88:0/2591153804 wait complete. 2024-02-19T22:17:32.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:32.101+0000 7fd874aec700 1 Processor -- start 2024-02-19T22:17:32.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:32.101+0000 7fd874aec700 1 -- start start 2024-02-19T22:17:32.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:32.101+0000 7fd874aec700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8700f4740 0x7fd8701067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:32.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:32.101+0000 7fd874aec700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd870106ca0 con 0x7fd8700f4740 2024-02-19T22:17:32.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:32.101+0000 7fd86e59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8700f4740 0x7fd8701067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:32.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:32.101+0000 7fd86e59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8700f4740 0x7fd8701067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:56544/0 (socket says 172.21.15.88:56544) 2024-02-19T22:17:32.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:32.101+0000 7fd86e59c700 1 -- 172.21.15.88:0/2042873293 learned_addr learned my addr 172.21.15.88:0/2042873293 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:17:32.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:32.101+0000 7fd86e59c700 1 -- 172.21.15.88:0/2042873293 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd864005040 con 0x7fd8700f4740 2024-02-19T22:17:32.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:32.102+0000 7fd86e59c700 1 --2- 172.21.15.88:0/2042873293 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8700f4740 0x7fd8701067b0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7fd864005010 tx=0x7fd86400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:32.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:32.102+0000 7fd85effd700 1 -- 172.21.15.88:0/2042873293 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd864009320 con 0x7fd8700f4740 2024-02-19T22:17:32.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:32.102+0000 7fd874aec700 1 -- 172.21.15.88:0/2042873293 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd870106ea0 con 0x7fd8700f4740 2024-02-19T22:17:32.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:32.102+0000 7fd874aec700 1 -- 172.21.15.88:0/2042873293 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd870107330 con 0x7fd8700f4740 2024-02-19T22:17:32.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:32.102+0000 7fd85effd700 1 -- 172.21.15.88:0/2042873293 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd864013380 con 0x7fd8700f4740 2024-02-19T22:17:32.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:32.102+0000 7fd85effd700 1 -- 172.21.15.88:0/2042873293 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd864014e50 con 0x7fd8700f4740 2024-02-19T22:17:32.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:32.103+0000 7fd85effd700 1 -- 172.21.15.88:0/2042873293 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd8640263a0 con 0x7fd8700f4740 2024-02-19T22:17:32.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:32.103+0000 7fd85effd700 1 --2- 172.21.15.88:0/2042873293 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd85802ff10 0x7fd8580323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:32.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:32.103+0000 7fd874aec700 1 -- 172.21.15.88:0/2042873293 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd870045ad0 con 0x7fd8700f4740 2024-02-19T22:17:32.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:32.103+0000 7fd85effd700 1 -- 172.21.15.88:0/2042873293 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd864040eb0 con 0x7fd8700f4740 2024-02-19T22:17:32.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:32.103+0000 7fd86dd9b700 1 --2- 172.21.15.88:0/2042873293 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd85802ff10 0x7fd8580323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:32.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:32.103+0000 7fd86dd9b700 1 --2- 172.21.15.88:0/2042873293 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd85802ff10 0x7fd8580323d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fd8600096a0 tx=0x7fd860006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:32.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:32.108+0000 7fd85effd700 1 -- 172.21.15.88:0/2042873293 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd864012510 con 0x7fd8700f4740 2024-02-19T22:17:32.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:32.313+0000 7fd874aec700 1 -- 172.21.15.88:0/2042873293 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd8700fc080 con 0x7fd8700f4740 2024-02-19T22:17:32.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:32.314+0000 7fd85effd700 1 -- 172.21.15.88:0/2042873293 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd864012330 con 0x7fd8700f4740 2024-02-19T22:17:32.316 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:17:32.316 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:32.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:32.315+0000 7fd874aec700 1 -- 172.21.15.88:0/2042873293 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd85802ff10 msgr2=0x7fd8580323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:32.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:32.315+0000 7fd874aec700 1 --2- 172.21.15.88:0/2042873293 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd85802ff10 0x7fd8580323d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fd8600096a0 tx=0x7fd860006b40).stop 2024-02-19T22:17:32.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:32.315+0000 7fd874aec700 1 -- 172.21.15.88:0/2042873293 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8700f4740 msgr2=0x7fd8701067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:32.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:32.315+0000 7fd874aec700 1 --2- 172.21.15.88:0/2042873293 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8700f4740 0x7fd8701067b0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7fd864005010 tx=0x7fd86400b040).stop 2024-02-19T22:17:32.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:32.315+0000 7fd874aec700 1 -- 172.21.15.88:0/2042873293 shutdown_connections 2024-02-19T22:17:32.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:32.315+0000 7fd874aec700 1 --2- 172.21.15.88:0/2042873293 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd85802ff10 0x7fd8580323d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:32.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:32.315+0000 7fd874aec700 1 --2- 172.21.15.88:0/2042873293 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8700f4740 0x7fd8701067b0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:32.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:32.315+0000 7fd874aec700 1 -- 172.21.15.88:0/2042873293 >> 172.21.15.88:0/2042873293 conn(0x7fd8700f0260 msgr2=0x7fd8700f0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:32.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:32.315+0000 7fd874aec700 1 -- 172.21.15.88:0/2042873293 shutdown_connections 2024-02-19T22:17:32.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:32.315+0000 7fd874aec700 1 -- 172.21.15.88:0/2042873293 wait complete. 2024-02-19T22:17:32.319 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:17:33.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:33 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:17:31.691419+0000 mgr.smithi052.ghqrtk (mgr.14162) 242 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:33.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:33 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:17:32.315215+0000 mon.smithi052 (mon.0) 256 : audit [DBG] from='client.? 172.21.15.88:0/2042873293' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:33.979 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:33.980 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:35.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:35 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:17:33.691777+0000 mgr.smithi052.ghqrtk (mgr.14162) 243 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:35.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:35.833+0000 7f0d5b884700 1 -- 172.21.15.88:0/1294266234 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0d54060220 msgr2=0x7f0d540fb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:35.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:35.833+0000 7f0d5b884700 1 --2- 172.21.15.88:0/1294266234 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0d54060220 0x7f0d540fb550 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f0d44009fb0 tx=0x7f0d44009b30).stop 2024-02-19T22:17:35.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:35.833+0000 7f0d5b884700 1 -- 172.21.15.88:0/1294266234 shutdown_connections 2024-02-19T22:17:35.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:35.833+0000 7f0d5b884700 1 --2- 172.21.15.88:0/1294266234 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0d54060220 0x7f0d540fb550 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:35.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:35.833+0000 7f0d5b884700 1 -- 172.21.15.88:0/1294266234 >> 172.21.15.88:0/1294266234 conn(0x7f0d540f0b70 msgr2=0x7f0d540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:35.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:35.833+0000 7f0d5b884700 1 -- 172.21.15.88:0/1294266234 shutdown_connections 2024-02-19T22:17:35.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:35.833+0000 7f0d5b884700 1 -- 172.21.15.88:0/1294266234 wait complete. 2024-02-19T22:17:35.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:35.834+0000 7f0d5b884700 1 Processor -- start 2024-02-19T22:17:35.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:35.834+0000 7f0d5b884700 1 -- start start 2024-02-19T22:17:35.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:35.834+0000 7f0d5b884700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0d54060220 0x7f0d541089a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:35.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:35.834+0000 7f0d5b884700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d54108e90 con 0x7f0d54060220 2024-02-19T22:17:35.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:35.834+0000 7f0d59620700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0d54060220 0x7f0d541089a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:35.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:35.834+0000 7f0d59620700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0d54060220 0x7f0d541089a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:48822/0 (socket says 172.21.15.88:48822) 2024-02-19T22:17:35.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:35.834+0000 7f0d59620700 1 -- 172.21.15.88:0/2510940454 learned_addr learned my addr 172.21.15.88:0/2510940454 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:17:35.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:35.834+0000 7f0d59620700 1 -- 172.21.15.88:0/2510940454 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d44005040 con 0x7f0d54060220 2024-02-19T22:17:35.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:35.834+0000 7f0d59620700 1 --2- 172.21.15.88:0/2510940454 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0d54060220 0x7f0d541089a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f0d440089f0 tx=0x7f0d4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:35.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:35.835+0000 7f0d49ffb700 1 -- 172.21.15.88:0/2510940454 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d44009320 con 0x7f0d54060220 2024-02-19T22:17:35.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:35.835+0000 7f0d5b884700 1 -- 172.21.15.88:0/2510940454 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d54109090 con 0x7f0d54060220 2024-02-19T22:17:35.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:35.835+0000 7f0d49ffb700 1 -- 172.21.15.88:0/2510940454 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0d44013380 con 0x7f0d54060220 2024-02-19T22:17:35.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:35.835+0000 7f0d49ffb700 1 -- 172.21.15.88:0/2510940454 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d44014e20 con 0x7f0d54060220 2024-02-19T22:17:35.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:35.835+0000 7f0d5b884700 1 -- 172.21.15.88:0/2510940454 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d54109520 con 0x7f0d54060220 2024-02-19T22:17:35.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:35.836+0000 7f0d49ffb700 1 -- 172.21.15.88:0/2510940454 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0d440263a0 con 0x7f0d54060220 2024-02-19T22:17:35.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:35.836+0000 7f0d49ffb700 1 --2- 172.21.15.88:0/2510940454 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f0d4002ff10 0x7f0d400323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:35.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:35.836+0000 7f0d49ffb700 1 -- 172.21.15.88:0/2510940454 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0d44040f60 con 0x7f0d54060220 2024-02-19T22:17:35.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:35.836+0000 7f0d5b884700 1 -- 172.21.15.88:0/2510940454 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d54045ad0 con 0x7f0d54060220 2024-02-19T22:17:35.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:35.836+0000 7f0d58e1f700 1 --2- 172.21.15.88:0/2510940454 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f0d4002ff10 0x7f0d400323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:35.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:35.836+0000 7f0d58e1f700 1 --2- 172.21.15.88:0/2510940454 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f0d4002ff10 0x7f0d400323d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f0d500096a0 tx=0x7f0d50006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:35.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:35.841+0000 7f0d49ffb700 1 -- 172.21.15.88:0/2510940454 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0d44012920 con 0x7f0d54060220 2024-02-19T22:17:36.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:36.048+0000 7f0d5b884700 1 -- 172.21.15.88:0/2510940454 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0d5410a260 con 0x7f0d54060220 2024-02-19T22:17:36.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:36.049+0000 7f0d49ffb700 1 -- 172.21.15.88:0/2510940454 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0d4402b020 con 0x7f0d54060220 2024-02-19T22:17:36.051 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:17:36.051 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:36.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:36.050+0000 7f0d5b884700 1 -- 172.21.15.88:0/2510940454 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f0d4002ff10 msgr2=0x7f0d400323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:36.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:36.050+0000 7f0d5b884700 1 --2- 172.21.15.88:0/2510940454 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f0d4002ff10 0x7f0d400323d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f0d500096a0 tx=0x7f0d50006b40).stop 2024-02-19T22:17:36.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:36.050+0000 7f0d5b884700 1 -- 172.21.15.88:0/2510940454 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0d54060220 msgr2=0x7f0d541089a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:36.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:36.050+0000 7f0d5b884700 1 --2- 172.21.15.88:0/2510940454 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0d54060220 0x7f0d541089a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f0d440089f0 tx=0x7f0d4400b040).stop 2024-02-19T22:17:36.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:36.050+0000 7f0d5b884700 1 -- 172.21.15.88:0/2510940454 shutdown_connections 2024-02-19T22:17:36.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:36.050+0000 7f0d5b884700 1 --2- 172.21.15.88:0/2510940454 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f0d4002ff10 0x7f0d400323d0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:36.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:36.050+0000 7f0d5b884700 1 --2- 172.21.15.88:0/2510940454 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0d54060220 0x7f0d541089a0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:36.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:36.050+0000 7f0d5b884700 1 -- 172.21.15.88:0/2510940454 >> 172.21.15.88:0/2510940454 conn(0x7f0d540f0b70 msgr2=0x7f0d540f1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:36.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:36.050+0000 7f0d5b884700 1 -- 172.21.15.88:0/2510940454 shutdown_connections 2024-02-19T22:17:36.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:36.050+0000 7f0d5b884700 1 -- 172.21.15.88:0/2510940454 wait complete. 2024-02-19T22:17:36.054 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:17:36.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:36 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:17:36.050179+0000 mon.smithi052 (mon.0) 257 : audit [DBG] from='client.? 172.21.15.88:0/2510940454' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:37.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:37 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:17:35.692180+0000 mgr.smithi052.ghqrtk (mgr.14162) 244 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:37.698 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:37.698 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:39.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:17:37.692511+0000 mgr.smithi052.ghqrtk (mgr.14162) 245 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:39.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:39.639+0000 7f3b42cec700 1 -- 172.21.15.88:0/3888830756 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3b3c0f5410 msgr2=0x7f3b3c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:39.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:39.639+0000 7f3b42cec700 1 --2- 172.21.15.88:0/3888830756 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3b3c0f5410 0x7f3b3c0f5830 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f3b30009fb0 tx=0x7f3b30009b30).stop 2024-02-19T22:17:39.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:39.639+0000 7f3b42cec700 1 -- 172.21.15.88:0/3888830756 shutdown_connections 2024-02-19T22:17:39.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:39.639+0000 7f3b42cec700 1 --2- 172.21.15.88:0/3888830756 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3b3c0f5410 0x7f3b3c0f5830 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:39.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:39.639+0000 7f3b42cec700 1 -- 172.21.15.88:0/3888830756 >> 172.21.15.88:0/3888830756 conn(0x7f3b3c0f0b70 msgr2=0x7f3b3c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:39.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:39.639+0000 7f3b42cec700 1 -- 172.21.15.88:0/3888830756 shutdown_connections 2024-02-19T22:17:39.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:39.639+0000 7f3b42cec700 1 -- 172.21.15.88:0/3888830756 wait complete. 2024-02-19T22:17:39.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:39.639+0000 7f3b42cec700 1 Processor -- start 2024-02-19T22:17:39.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:39.639+0000 7f3b42cec700 1 -- start start 2024-02-19T22:17:39.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:39.640+0000 7f3b42cec700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3b3c0f5410 0x7f3b3c1094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:39.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:39.640+0000 7f3b42cec700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3c060240 con 0x7f3b3c0f5410 2024-02-19T22:17:39.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:39.640+0000 7f3b40a88700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3b3c0f5410 0x7f3b3c1094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:39.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:39.640+0000 7f3b40a88700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3b3c0f5410 0x7f3b3c1094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:48850/0 (socket says 172.21.15.88:48850) 2024-02-19T22:17:39.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:39.640+0000 7f3b40a88700 1 -- 172.21.15.88:0/32759472 learned_addr learned my addr 172.21.15.88:0/32759472 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:17:39.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:39.640+0000 7f3b40a88700 1 -- 172.21.15.88:0/32759472 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b30005040 con 0x7f3b3c0f5410 2024-02-19T22:17:39.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:39.640+0000 7f3b40a88700 1 --2- 172.21.15.88:0/32759472 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3b3c0f5410 0x7f3b3c1094e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f3b30005010 tx=0x7f3b3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:39.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:39.640+0000 7f3b397fa700 1 -- 172.21.15.88:0/32759472 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b30009320 con 0x7f3b3c0f5410 2024-02-19T22:17:39.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:39.640+0000 7f3b42cec700 1 -- 172.21.15.88:0/32759472 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b3c060440 con 0x7f3b3c0f5410 2024-02-19T22:17:39.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:39.641+0000 7f3b397fa700 1 -- 172.21.15.88:0/32759472 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3b30013380 con 0x7f3b3c0f5410 2024-02-19T22:17:39.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:39.641+0000 7f3b397fa700 1 -- 172.21.15.88:0/32759472 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b30014e50 con 0x7f3b3c0f5410 2024-02-19T22:17:39.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:39.641+0000 7f3b42cec700 1 -- 172.21.15.88:0/32759472 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b3c109b90 con 0x7f3b3c0f5410 2024-02-19T22:17:39.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:39.641+0000 7f3b397fa700 1 -- 172.21.15.88:0/32759472 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3b300263a0 con 0x7f3b3c0f5410 2024-02-19T22:17:39.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:39.641+0000 7f3b397fa700 1 --2- 172.21.15.88:0/32759472 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f3b2c02ff10 0x7f3b2c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:39.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:39.641+0000 7f3b397fa700 1 -- 172.21.15.88:0/32759472 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3b30040ec0 con 0x7f3b3c0f5410 2024-02-19T22:17:39.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:39.641+0000 7f3b42cec700 1 -- 172.21.15.88:0/32759472 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b3c045ad0 con 0x7f3b3c0f5410 2024-02-19T22:17:39.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:39.642+0000 7f3b3bfff700 1 --2- 172.21.15.88:0/32759472 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f3b2c02ff10 0x7f3b2c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:39.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:39.642+0000 7f3b3bfff700 1 --2- 172.21.15.88:0/32759472 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f3b2c02ff10 0x7f3b2c0323d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f3b280096a0 tx=0x7f3b28006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:39.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:39.646+0000 7f3b397fa700 1 -- 172.21.15.88:0/32759472 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3b30012920 con 0x7f3b3c0f5410 2024-02-19T22:17:39.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:39.850+0000 7f3b42cec700 1 -- 172.21.15.88:0/32759472 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3b3c10a260 con 0x7f3b3c0f5410 2024-02-19T22:17:39.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:39.850+0000 7f3b397fa700 1 -- 172.21.15.88:0/32759472 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3b3002b020 con 0x7f3b3c0f5410 2024-02-19T22:17:39.853 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:17:39.853 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:39.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:39.852+0000 7f3b42cec700 1 -- 172.21.15.88:0/32759472 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f3b2c02ff10 msgr2=0x7f3b2c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:39.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:39.852+0000 7f3b42cec700 1 --2- 172.21.15.88:0/32759472 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f3b2c02ff10 0x7f3b2c0323d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f3b280096a0 tx=0x7f3b28006b40).stop 2024-02-19T22:17:39.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:39.852+0000 7f3b42cec700 1 -- 172.21.15.88:0/32759472 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3b3c0f5410 msgr2=0x7f3b3c1094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:39.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:39.852+0000 7f3b42cec700 1 --2- 172.21.15.88:0/32759472 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3b3c0f5410 0x7f3b3c1094e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f3b30005010 tx=0x7f3b3000b040).stop 2024-02-19T22:17:39.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:39.852+0000 7f3b42cec700 1 -- 172.21.15.88:0/32759472 shutdown_connections 2024-02-19T22:17:39.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:39.852+0000 7f3b42cec700 1 --2- 172.21.15.88:0/32759472 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f3b2c02ff10 0x7f3b2c0323d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:39.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:39.852+0000 7f3b42cec700 1 --2- 172.21.15.88:0/32759472 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3b3c0f5410 0x7f3b3c1094e0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:39.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:39.852+0000 7f3b42cec700 1 -- 172.21.15.88:0/32759472 >> 172.21.15.88:0/32759472 conn(0x7f3b3c0f0b70 msgr2=0x7f3b3c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:39.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:39.852+0000 7f3b42cec700 1 -- 172.21.15.88:0/32759472 shutdown_connections 2024-02-19T22:17:39.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:39.852+0000 7f3b42cec700 1 -- 172.21.15.88:0/32759472 wait complete. 2024-02-19T22:17:39.855 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:17:40.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:40 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:17:39.851728+0000 mon.smithi052 (mon.0) 258 : audit [DBG] from='client.? 172.21.15.88:0/32759472' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:41.333 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:41.333 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:41.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:41 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:17:39.692869+0000 mgr.smithi052.ghqrtk (mgr.14162) 246 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:43.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:43.326+0000 7fe9453c3700 1 -- 172.21.15.88:0/289044763 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe9400f51a0 msgr2=0x7fe9400f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:43.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:43.326+0000 7fe9453c3700 1 --2- 172.21.15.88:0/289044763 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe9400f51a0 0x7fe9400f55c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7fe930009fb0 tx=0x7fe930009b30).stop 2024-02-19T22:17:43.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:43.326+0000 7fe9453c3700 1 -- 172.21.15.88:0/289044763 shutdown_connections 2024-02-19T22:17:43.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:43.326+0000 7fe9453c3700 1 --2- 172.21.15.88:0/289044763 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe9400f51a0 0x7fe9400f55c0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:43.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:43.326+0000 7fe9453c3700 1 -- 172.21.15.88:0/289044763 >> 172.21.15.88:0/289044763 conn(0x7fe9400f09a0 msgr2=0x7fe9400f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:43.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:43.326+0000 7fe9453c3700 1 -- 172.21.15.88:0/289044763 shutdown_connections 2024-02-19T22:17:43.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:43.326+0000 7fe9453c3700 1 -- 172.21.15.88:0/289044763 wait complete. 2024-02-19T22:17:43.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:43.326+0000 7fe9453c3700 1 Processor -- start 2024-02-19T22:17:43.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:43.327+0000 7fe9453c3700 1 -- start start 2024-02-19T22:17:43.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:43.327+0000 7fe9453c3700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe9400f51a0 0x7fe9400fd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:43.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:43.327+0000 7fe9453c3700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9400fb6f0 con 0x7fe9400f51a0 2024-02-19T22:17:43.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:43.327+0000 7fe93ffff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe9400f51a0 0x7fe9400fd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:43.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:43.327+0000 7fe93ffff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe9400f51a0 0x7fe9400fd270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:39528/0 (socket says 172.21.15.88:39528) 2024-02-19T22:17:43.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:43.327+0000 7fe93ffff700 1 -- 172.21.15.88:0/3692273432 learned_addr learned my addr 172.21.15.88:0/3692273432 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:17:43.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:43.327+0000 7fe93ffff700 1 -- 172.21.15.88:0/3692273432 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe930005040 con 0x7fe9400f51a0 2024-02-19T22:17:43.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:43.327+0000 7fe93ffff700 1 --2- 172.21.15.88:0/3692273432 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe9400f51a0 0x7fe9400fd270 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fe9300089f0 tx=0x7fe93000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:43.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:43.327+0000 7fe93cff9700 1 -- 172.21.15.88:0/3692273432 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe930009320 con 0x7fe9400f51a0 2024-02-19T22:17:43.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:43.327+0000 7fe9453c3700 1 -- 172.21.15.88:0/3692273432 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9400fb890 con 0x7fe9400f51a0 2024-02-19T22:17:43.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:43.327+0000 7fe93cff9700 1 -- 172.21.15.88:0/3692273432 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe930013380 con 0x7fe9400f51a0 2024-02-19T22:17:43.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:43.327+0000 7fe93cff9700 1 -- 172.21.15.88:0/3692273432 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe930014e20 con 0x7fe9400f51a0 2024-02-19T22:17:43.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:43.328+0000 7fe9453c3700 1 -- 172.21.15.88:0/3692273432 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9400fbd20 con 0x7fe9400f51a0 2024-02-19T22:17:43.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:43.328+0000 7fe93cff9700 1 -- 172.21.15.88:0/3692273432 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe930014420 con 0x7fe9400f51a0 2024-02-19T22:17:43.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:43.328+0000 7fe93cff9700 1 --2- 172.21.15.88:0/3692273432 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fe92802ff10 0x7fe9280323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:43.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:43.328+0000 7fe9453c3700 1 -- 172.21.15.88:0/3692273432 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe940059100 con 0x7fe9400f51a0 2024-02-19T22:17:43.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:43.329+0000 7fe93cff9700 1 -- 172.21.15.88:0/3692273432 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe930041010 con 0x7fe9400f51a0 2024-02-19T22:17:43.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:43.329+0000 7fe93f7fe700 1 --2- 172.21.15.88:0/3692273432 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fe92802ff10 0x7fe9280323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:43.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:43.329+0000 7fe93f7fe700 1 --2- 172.21.15.88:0/3692273432 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fe92802ff10 0x7fe9280323d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fe9340096a0 tx=0x7fe934006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:43.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:43.333+0000 7fe93cff9700 1 -- 172.21.15.88:0/3692273432 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe9300095f0 con 0x7fe9400f51a0 2024-02-19T22:17:43.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:43.540+0000 7fe9453c3700 1 -- 172.21.15.88:0/3692273432 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe9400fad30 con 0x7fe9400f51a0 2024-02-19T22:17:43.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:43.540+0000 7fe93cff9700 1 -- 172.21.15.88:0/3692273432 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe93001db70 con 0x7fe9400f51a0 2024-02-19T22:17:43.543 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:17:43.543 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:43.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:43.542+0000 7fe9453c3700 1 -- 172.21.15.88:0/3692273432 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fe92802ff10 msgr2=0x7fe9280323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:43.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:43.542+0000 7fe9453c3700 1 --2- 172.21.15.88:0/3692273432 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fe92802ff10 0x7fe9280323d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fe9340096a0 tx=0x7fe934006b40).stop 2024-02-19T22:17:43.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:43.542+0000 7fe9453c3700 1 -- 172.21.15.88:0/3692273432 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe9400f51a0 msgr2=0x7fe9400fd270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:43.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:43.542+0000 7fe9453c3700 1 --2- 172.21.15.88:0/3692273432 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe9400f51a0 0x7fe9400fd270 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fe9300089f0 tx=0x7fe93000b040).stop 2024-02-19T22:17:43.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:43.542+0000 7fe9453c3700 1 -- 172.21.15.88:0/3692273432 shutdown_connections 2024-02-19T22:17:43.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:43.542+0000 7fe9453c3700 1 --2- 172.21.15.88:0/3692273432 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fe92802ff10 0x7fe9280323d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:43.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:43.542+0000 7fe9453c3700 1 --2- 172.21.15.88:0/3692273432 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe9400f51a0 0x7fe9400fd270 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:43.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:43.542+0000 7fe9453c3700 1 -- 172.21.15.88:0/3692273432 >> 172.21.15.88:0/3692273432 conn(0x7fe9400f09a0 msgr2=0x7fe94005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:43.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:43.542+0000 7fe9453c3700 1 -- 172.21.15.88:0/3692273432 shutdown_connections 2024-02-19T22:17:43.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:43.542+0000 7fe9453c3700 1 -- 172.21.15.88:0/3692273432 wait complete. 2024-02-19T22:17:43.545 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:17:43.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:43 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:17:41.693185+0000 mgr.smithi052.ghqrtk (mgr.14162) 247 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:44.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:44 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:17:43.542049+0000 mon.smithi052 (mon.0) 259 : audit [DBG] from='client.? 172.21.15.88:0/3692273432' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:44.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:44 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:17:43.752941+0000 mon.smithi052 (mon.0) 260 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi052.ghqrtk/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:17:44.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:44 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:17:43.755728+0000 mon.smithi052 (mon.0) 261 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi052.ghqrtk/trash_purge_schedule"}]: dispatch 2024-02-19T22:17:45.177 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:45.178 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:45.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:45 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:17:43.693415+0000 mgr.smithi052.ghqrtk (mgr.14162) 248 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:47.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:47.128+0000 7f0950854700 1 -- 172.21.15.88:0/1471550324 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f094c0f62a0 msgr2=0x7f094c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:47.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:47.128+0000 7f0950854700 1 --2- 172.21.15.88:0/1471550324 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f094c0f62a0 0x7f094c0f66c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f0940009fb0 tx=0x7f0940009b30).stop 2024-02-19T22:17:47.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:47.128+0000 7f0950854700 1 -- 172.21.15.88:0/1471550324 shutdown_connections 2024-02-19T22:17:47.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:47.128+0000 7f0950854700 1 --2- 172.21.15.88:0/1471550324 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f094c0f62a0 0x7f094c0f66c0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:47.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:47.128+0000 7f0950854700 1 -- 172.21.15.88:0/1471550324 >> 172.21.15.88:0/1471550324 conn(0x7f094c0f1aa0 msgr2=0x7f094c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:47.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:47.128+0000 7f0950854700 1 -- 172.21.15.88:0/1471550324 shutdown_connections 2024-02-19T22:17:47.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:47.128+0000 7f0950854700 1 -- 172.21.15.88:0/1471550324 wait complete. 2024-02-19T22:17:47.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:47.129+0000 7f0950854700 1 Processor -- start 2024-02-19T22:17:47.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:47.129+0000 7f0950854700 1 -- start start 2024-02-19T22:17:47.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:47.129+0000 7f0950854700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f094c0f62a0 0x7f094c1065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:47.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:47.129+0000 7f0950854700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f094c106ac0 con 0x7f094c0f62a0 2024-02-19T22:17:47.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:47.130+0000 7f094ad9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f094c0f62a0 0x7f094c1065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:47.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:47.130+0000 7f094ad9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f094c0f62a0 0x7f094c1065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:39540/0 (socket says 172.21.15.88:39540) 2024-02-19T22:17:47.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:47.130+0000 7f094ad9d700 1 -- 172.21.15.88:0/292427833 learned_addr learned my addr 172.21.15.88:0/292427833 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:17:47.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:47.130+0000 7f094ad9d700 1 -- 172.21.15.88:0/292427833 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0940005040 con 0x7f094c0f62a0 2024-02-19T22:17:47.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:47.130+0000 7f094ad9d700 1 --2- 172.21.15.88:0/292427833 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f094c0f62a0 0x7f094c1065d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f09400089f0 tx=0x7f094000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:47.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:47.130+0000 7f09337fe700 1 -- 172.21.15.88:0/292427833 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0940009320 con 0x7f094c0f62a0 2024-02-19T22:17:47.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:47.130+0000 7f0950854700 1 -- 172.21.15.88:0/292427833 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f094c106cc0 con 0x7f094c0f62a0 2024-02-19T22:17:47.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:47.130+0000 7f09337fe700 1 -- 172.21.15.88:0/292427833 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0940013380 con 0x7f094c0f62a0 2024-02-19T22:17:47.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:47.130+0000 7f09337fe700 1 -- 172.21.15.88:0/292427833 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0940014e20 con 0x7f094c0f62a0 2024-02-19T22:17:47.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:47.130+0000 7f0950854700 1 -- 172.21.15.88:0/292427833 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f094c107150 con 0x7f094c0f62a0 2024-02-19T22:17:47.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:47.131+0000 7f09337fe700 1 -- 172.21.15.88:0/292427833 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0940014420 con 0x7f094c0f62a0 2024-02-19T22:17:47.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:47.131+0000 7f0950854700 1 -- 172.21.15.88:0/292427833 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f094c045ad0 con 0x7f094c0f62a0 2024-02-19T22:17:47.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:47.131+0000 7f09337fe700 1 --2- 172.21.15.88:0/292427833 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f093402ff10 0x7f09340323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:47.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:47.131+0000 7f09337fe700 1 -- 172.21.15.88:0/292427833 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0940040f50 con 0x7f094c0f62a0 2024-02-19T22:17:47.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:47.131+0000 7f094a59c700 1 --2- 172.21.15.88:0/292427833 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f093402ff10 0x7f09340323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:47.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:47.132+0000 7f094a59c700 1 --2- 172.21.15.88:0/292427833 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f093402ff10 0x7f09340323d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f093c0096a0 tx=0x7f093c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:47.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:47.137+0000 7f09337fe700 1 -- 172.21.15.88:0/292427833 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f094006d2b0 con 0x7f094c0f62a0 2024-02-19T22:17:47.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:47.340+0000 7f0950854700 1 -- 172.21.15.88:0/292427833 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f094c108000 con 0x7f094c0f62a0 2024-02-19T22:17:47.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:47.341+0000 7f09337fe700 1 -- 172.21.15.88:0/292427833 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f09400095f0 con 0x7f094c0f62a0 2024-02-19T22:17:47.344 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:17:47.344 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:47.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:47.342+0000 7f0950854700 1 -- 172.21.15.88:0/292427833 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f093402ff10 msgr2=0x7f09340323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:47.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:47.342+0000 7f0950854700 1 --2- 172.21.15.88:0/292427833 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f093402ff10 0x7f09340323d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f093c0096a0 tx=0x7f093c006b40).stop 2024-02-19T22:17:47.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:47.342+0000 7f0950854700 1 -- 172.21.15.88:0/292427833 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f094c0f62a0 msgr2=0x7f094c1065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:47.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:47.342+0000 7f0950854700 1 --2- 172.21.15.88:0/292427833 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f094c0f62a0 0x7f094c1065d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f09400089f0 tx=0x7f094000b040).stop 2024-02-19T22:17:47.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:47.342+0000 7f0950854700 1 -- 172.21.15.88:0/292427833 shutdown_connections 2024-02-19T22:17:47.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:47.342+0000 7f0950854700 1 --2- 172.21.15.88:0/292427833 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f093402ff10 0x7f09340323d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:47.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:47.342+0000 7f0950854700 1 --2- 172.21.15.88:0/292427833 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f094c0f62a0 0x7f094c1065d0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:47.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:47.342+0000 7f0950854700 1 -- 172.21.15.88:0/292427833 >> 172.21.15.88:0/292427833 conn(0x7f094c0f1aa0 msgr2=0x7f094c0f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:47.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:47.342+0000 7f0950854700 1 -- 172.21.15.88:0/292427833 shutdown_connections 2024-02-19T22:17:47.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:47.342+0000 7f0950854700 1 -- 172.21.15.88:0/292427833 wait complete. 2024-02-19T22:17:47.346 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:17:47.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:47 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:17:45.693665+0000 mgr.smithi052.ghqrtk (mgr.14162) 249 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:48.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:48 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:17:47.342381+0000 mon.smithi052 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.88:0/292427833' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:48.949 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:48.949 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:49.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:49 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:17:47.693946+0000 mgr.smithi052.ghqrtk (mgr.14162) 250 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:50.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:50.771+0000 7fb5833cb700 1 -- 172.21.15.88:0/630684645 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb57c060280 msgr2=0x7fb57c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:50.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:50.771+0000 7fb5833cb700 1 --2- 172.21.15.88:0/630684645 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb57c060280 0x7fb57c0f90f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7fb578009fb0 tx=0x7fb578009b30).stop 2024-02-19T22:17:50.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:50.771+0000 7fb5833cb700 1 -- 172.21.15.88:0/630684645 shutdown_connections 2024-02-19T22:17:50.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:50.771+0000 7fb5833cb700 1 --2- 172.21.15.88:0/630684645 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb57c060280 0x7fb57c0f90f0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:50.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:50.771+0000 7fb5833cb700 1 -- 172.21.15.88:0/630684645 >> 172.21.15.88:0/630684645 conn(0x7fb57c0f0b70 msgr2=0x7fb57c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:50.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:50.771+0000 7fb5833cb700 1 -- 172.21.15.88:0/630684645 shutdown_connections 2024-02-19T22:17:50.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:50.771+0000 7fb5833cb700 1 -- 172.21.15.88:0/630684645 wait complete. 2024-02-19T22:17:50.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:50.771+0000 7fb5833cb700 1 Processor -- start 2024-02-19T22:17:50.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:50.772+0000 7fb5833cb700 1 -- start start 2024-02-19T22:17:50.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:50.772+0000 7fb5833cb700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb57c060280 0x7fb57c0f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:50.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:50.772+0000 7fb5833cb700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb57c0f4ff0 con 0x7fb57c060280 2024-02-19T22:17:50.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:50.772+0000 7fb581167700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb57c060280 0x7fb57c0f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:50.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:50.772+0000 7fb581167700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb57c060280 0x7fb57c0f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:39552/0 (socket says 172.21.15.88:39552) 2024-02-19T22:17:50.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:50.772+0000 7fb581167700 1 -- 172.21.15.88:0/1215080531 learned_addr learned my addr 172.21.15.88:0/1215080531 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:17:50.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:50.772+0000 7fb581167700 1 -- 172.21.15.88:0/1215080531 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb578005040 con 0x7fb57c060280 2024-02-19T22:17:50.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:50.772+0000 7fb581167700 1 --2- 172.21.15.88:0/1215080531 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb57c060280 0x7fb57c0f6b70 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7fb5780089f0 tx=0x7fb57800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:50.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:50.772+0000 7fb571ffb700 1 -- 172.21.15.88:0/1215080531 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb578009320 con 0x7fb57c060280 2024-02-19T22:17:50.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:50.772+0000 7fb5833cb700 1 -- 172.21.15.88:0/1215080531 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb57c0f5190 con 0x7fb57c060280 2024-02-19T22:17:50.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:50.773+0000 7fb5833cb700 1 -- 172.21.15.88:0/1215080531 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb57c0f5620 con 0x7fb57c060280 2024-02-19T22:17:50.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:50.773+0000 7fb571ffb700 1 -- 172.21.15.88:0/1215080531 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb578013380 con 0x7fb57c060280 2024-02-19T22:17:50.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:50.773+0000 7fb571ffb700 1 -- 172.21.15.88:0/1215080531 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb578014db0 con 0x7fb57c060280 2024-02-19T22:17:50.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:50.773+0000 7fb571ffb700 1 -- 172.21.15.88:0/1215080531 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb5780263a0 con 0x7fb57c060280 2024-02-19T22:17:50.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:50.773+0000 7fb5833cb700 1 -- 172.21.15.88:0/1215080531 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb55c004fa0 con 0x7fb57c060280 2024-02-19T22:17:50.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:50.774+0000 7fb571ffb700 1 --2- 172.21.15.88:0/1215080531 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fb56802ff60 0x7fb568032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:50.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:50.774+0000 7fb571ffb700 1 -- 172.21.15.88:0/1215080531 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb578041190 con 0x7fb57c060280 2024-02-19T22:17:50.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:50.774+0000 7fb580966700 1 --2- 172.21.15.88:0/1215080531 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fb56802ff60 0x7fb568032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:50.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:50.774+0000 7fb580966700 1 --2- 172.21.15.88:0/1215080531 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fb56802ff60 0x7fb568032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fb56c0096a0 tx=0x7fb56c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:50.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:50.778+0000 7fb571ffb700 1 -- 172.21.15.88:0/1215080531 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb5780126f0 con 0x7fb57c060280 2024-02-19T22:17:50.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:50.989+0000 7fb5833cb700 1 -- 172.21.15.88:0/1215080531 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb55c005670 con 0x7fb57c060280 2024-02-19T22:17:50.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:50.989+0000 7fb571ffb700 1 -- 172.21.15.88:0/1215080531 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb578004490 con 0x7fb57c060280 2024-02-19T22:17:50.991 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:17:50.991 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:50.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:50.991+0000 7fb5833cb700 1 -- 172.21.15.88:0/1215080531 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fb56802ff60 msgr2=0x7fb568032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:50.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:50.991+0000 7fb5833cb700 1 --2- 172.21.15.88:0/1215080531 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fb56802ff60 0x7fb568032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fb56c0096a0 tx=0x7fb56c006b40).stop 2024-02-19T22:17:50.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:50.991+0000 7fb5833cb700 1 -- 172.21.15.88:0/1215080531 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb57c060280 msgr2=0x7fb57c0f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:50.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:50.991+0000 7fb5833cb700 1 --2- 172.21.15.88:0/1215080531 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb57c060280 0x7fb57c0f6b70 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7fb5780089f0 tx=0x7fb57800b040).stop 2024-02-19T22:17:50.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:50.991+0000 7fb5833cb700 1 -- 172.21.15.88:0/1215080531 shutdown_connections 2024-02-19T22:17:50.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:50.991+0000 7fb5833cb700 1 --2- 172.21.15.88:0/1215080531 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fb56802ff60 0x7fb568032420 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:50.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:50.991+0000 7fb5833cb700 1 --2- 172.21.15.88:0/1215080531 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb57c060280 0x7fb57c0f6b70 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:50.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:50.991+0000 7fb5833cb700 1 -- 172.21.15.88:0/1215080531 >> 172.21.15.88:0/1215080531 conn(0x7fb57c0f0b70 msgr2=0x7fb57c0f15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:50.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:50.991+0000 7fb5833cb700 1 -- 172.21.15.88:0/1215080531 shutdown_connections 2024-02-19T22:17:50.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:50.991+0000 7fb5833cb700 1 -- 172.21.15.88:0/1215080531 wait complete. 2024-02-19T22:17:50.994 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:17:51.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:51 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:17:49.694191+0000 mgr.smithi052.ghqrtk (mgr.14162) 251 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:51.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:51 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:17:50.989872+0000 mon.smithi052 (mon.0) 263 : audit [DBG] from='client.? 172.21.15.88:0/1215080531' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:52.607 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:52.608 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:53.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:53 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:17:51.694408+0000 mgr.smithi052.ghqrtk (mgr.14162) 252 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:54.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:54.535+0000 7f4e5ff8a700 1 -- 172.21.15.88:0/2945009262 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4e58060280 msgr2=0x7f4e580f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:54.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:54.535+0000 7f4e5ff8a700 1 --2- 172.21.15.88:0/2945009262 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4e58060280 0x7f4e580f90f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f4e48009fb0 tx=0x7f4e48009b30).stop 2024-02-19T22:17:54.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:54.535+0000 7f4e5ff8a700 1 -- 172.21.15.88:0/2945009262 shutdown_connections 2024-02-19T22:17:54.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:54.535+0000 7f4e5ff8a700 1 --2- 172.21.15.88:0/2945009262 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4e58060280 0x7f4e580f90f0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:54.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:54.535+0000 7f4e5ff8a700 1 -- 172.21.15.88:0/2945009262 >> 172.21.15.88:0/2945009262 conn(0x7f4e580f0b70 msgr2=0x7f4e580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:54.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:54.535+0000 7f4e5ff8a700 1 -- 172.21.15.88:0/2945009262 shutdown_connections 2024-02-19T22:17:54.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:54.535+0000 7f4e5ff8a700 1 -- 172.21.15.88:0/2945009262 wait complete. 2024-02-19T22:17:54.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:54.536+0000 7f4e5ff8a700 1 Processor -- start 2024-02-19T22:17:54.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:54.536+0000 7f4e5ff8a700 1 -- start start 2024-02-19T22:17:54.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:54.536+0000 7f4e5ff8a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4e58060280 0x7f4e580f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:54.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:54.536+0000 7f4e5ff8a700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e580f4ff0 con 0x7f4e58060280 2024-02-19T22:17:54.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:54.536+0000 7f4e5dd26700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4e58060280 0x7f4e580f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:54.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:54.536+0000 7f4e5dd26700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4e58060280 0x7f4e580f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:39668/0 (socket says 172.21.15.88:39668) 2024-02-19T22:17:54.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:54.536+0000 7f4e5dd26700 1 -- 172.21.15.88:0/2033210017 learned_addr learned my addr 172.21.15.88:0/2033210017 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:17:54.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:54.536+0000 7f4e5dd26700 1 -- 172.21.15.88:0/2033210017 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e48005040 con 0x7f4e58060280 2024-02-19T22:17:54.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:54.537+0000 7f4e5dd26700 1 --2- 172.21.15.88:0/2033210017 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4e58060280 0x7f4e580f6b70 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f4e480089f0 tx=0x7f4e4800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:54.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:54.537+0000 7f4e4e7fc700 1 -- 172.21.15.88:0/2033210017 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e48009320 con 0x7f4e58060280 2024-02-19T22:17:54.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:54.537+0000 7f4e5ff8a700 1 -- 172.21.15.88:0/2033210017 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e580f5190 con 0x7f4e58060280 2024-02-19T22:17:54.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:54.537+0000 7f4e4e7fc700 1 -- 172.21.15.88:0/2033210017 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4e48013380 con 0x7f4e58060280 2024-02-19T22:17:54.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:54.537+0000 7f4e5ff8a700 1 -- 172.21.15.88:0/2033210017 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e580f5620 con 0x7f4e58060280 2024-02-19T22:17:54.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:54.537+0000 7f4e4e7fc700 1 -- 172.21.15.88:0/2033210017 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e48014e20 con 0x7f4e58060280 2024-02-19T22:17:54.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:54.538+0000 7f4e5ff8a700 1 -- 172.21.15.88:0/2033210017 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e58059100 con 0x7f4e58060280 2024-02-19T22:17:54.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:54.538+0000 7f4e4e7fc700 1 -- 172.21.15.88:0/2033210017 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4e48014420 con 0x7f4e58060280 2024-02-19T22:17:54.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:54.538+0000 7f4e4e7fc700 1 --2- 172.21.15.88:0/2033210017 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f4e4402ff10 0x7f4e440323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:54.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:54.538+0000 7f4e4e7fc700 1 -- 172.21.15.88:0/2033210017 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4e48040f90 con 0x7f4e58060280 2024-02-19T22:17:54.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:54.538+0000 7f4e5d525700 1 --2- 172.21.15.88:0/2033210017 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f4e4402ff10 0x7f4e440323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:54.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:54.538+0000 7f4e5d525700 1 --2- 172.21.15.88:0/2033210017 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f4e4402ff10 0x7f4e440323d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f4e540096a0 tx=0x7f4e54006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:54.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:54.543+0000 7f4e4e7fc700 1 -- 172.21.15.88:0/2033210017 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4e48014790 con 0x7f4e58060280 2024-02-19T22:17:54.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:54.743+0000 7f4e5ff8a700 1 -- 172.21.15.88:0/2033210017 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4e58045ad0 con 0x7f4e58060280 2024-02-19T22:17:54.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:54.744+0000 7f4e4e7fc700 1 -- 172.21.15.88:0/2033210017 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4e4801d7f0 con 0x7f4e58060280 2024-02-19T22:17:54.746 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:17:54.746 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:54.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:54.745+0000 7f4e5ff8a700 1 -- 172.21.15.88:0/2033210017 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f4e4402ff10 msgr2=0x7f4e440323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:54.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:54.745+0000 7f4e5ff8a700 1 --2- 172.21.15.88:0/2033210017 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f4e4402ff10 0x7f4e440323d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f4e540096a0 tx=0x7f4e54006b40).stop 2024-02-19T22:17:54.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:54.745+0000 7f4e5ff8a700 1 -- 172.21.15.88:0/2033210017 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4e58060280 msgr2=0x7f4e580f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:54.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:54.745+0000 7f4e5ff8a700 1 --2- 172.21.15.88:0/2033210017 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4e58060280 0x7f4e580f6b70 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f4e480089f0 tx=0x7f4e4800b040).stop 2024-02-19T22:17:54.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:54.745+0000 7f4e5ff8a700 1 -- 172.21.15.88:0/2033210017 shutdown_connections 2024-02-19T22:17:54.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:54.745+0000 7f4e5ff8a700 1 --2- 172.21.15.88:0/2033210017 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f4e4402ff10 0x7f4e440323d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:54.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:54.745+0000 7f4e5ff8a700 1 --2- 172.21.15.88:0/2033210017 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4e58060280 0x7f4e580f6b70 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:54.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:54.745+0000 7f4e5ff8a700 1 -- 172.21.15.88:0/2033210017 >> 172.21.15.88:0/2033210017 conn(0x7f4e580f0b70 msgr2=0x7f4e580f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:54.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:54.746+0000 7f4e5ff8a700 1 -- 172.21.15.88:0/2033210017 shutdown_connections 2024-02-19T22:17:54.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:54.746+0000 7f4e5ff8a700 1 -- 172.21.15.88:0/2033210017 wait complete. 2024-02-19T22:17:54.748 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:17:55.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:55 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:17:53.694651+0000 mgr.smithi052.ghqrtk (mgr.14162) 253 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:55.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:55 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:17:54.744533+0000 mon.smithi052 (mon.0) 264 : audit [DBG] from='client.? 172.21.15.88:0/2033210017' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:56.451 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:56.452 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:57.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:57 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:17:55.694990+0000 mgr.smithi052.ghqrtk (mgr.14162) 254 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:58.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:58.232+0000 7f51ca0a6700 1 -- 172.21.15.88:0/3958110827 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f51c40f51a0 msgr2=0x7f51c40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:58.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:58.232+0000 7f51ca0a6700 1 --2- 172.21.15.88:0/3958110827 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f51c40f51a0 0x7f51c40f55c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f51b4009fb0 tx=0x7f51b4009b30).stop 2024-02-19T22:17:58.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:58.232+0000 7f51ca0a6700 1 -- 172.21.15.88:0/3958110827 shutdown_connections 2024-02-19T22:17:58.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:58.232+0000 7f51ca0a6700 1 --2- 172.21.15.88:0/3958110827 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f51c40f51a0 0x7f51c40f55c0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:58.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:58.233+0000 7f51ca0a6700 1 -- 172.21.15.88:0/3958110827 >> 172.21.15.88:0/3958110827 conn(0x7f51c40f09a0 msgr2=0x7f51c40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:58.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:58.233+0000 7f51ca0a6700 1 -- 172.21.15.88:0/3958110827 shutdown_connections 2024-02-19T22:17:58.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:58.233+0000 7f51ca0a6700 1 -- 172.21.15.88:0/3958110827 wait complete. 2024-02-19T22:17:58.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:58.233+0000 7f51ca0a6700 1 Processor -- start 2024-02-19T22:17:58.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:58.233+0000 7f51ca0a6700 1 -- start start 2024-02-19T22:17:58.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:58.233+0000 7f51ca0a6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f51c40f51a0 0x7f51c41054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:58.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:58.233+0000 7f51ca0a6700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c41059c0 con 0x7f51c40f51a0 2024-02-19T22:17:58.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:58.234+0000 7f51c90a4700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f51c40f51a0 0x7f51c41054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:58.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:58.234+0000 7f51c90a4700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f51c40f51a0 0x7f51c41054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:39688/0 (socket says 172.21.15.88:39688) 2024-02-19T22:17:58.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:58.234+0000 7f51c90a4700 1 -- 172.21.15.88:0/3118508021 learned_addr learned my addr 172.21.15.88:0/3118508021 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:17:58.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:58.234+0000 7f51c90a4700 1 -- 172.21.15.88:0/3118508021 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51b4005040 con 0x7f51c40f51a0 2024-02-19T22:17:58.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:58.234+0000 7f51c90a4700 1 --2- 172.21.15.88:0/3118508021 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f51c40f51a0 0x7f51c41054d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f51b40089f0 tx=0x7f51b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:58.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:58.234+0000 7f51b9ffb700 1 -- 172.21.15.88:0/3118508021 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f51b4009320 con 0x7f51c40f51a0 2024-02-19T22:17:58.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:58.234+0000 7f51b9ffb700 1 -- 172.21.15.88:0/3118508021 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f51b4013380 con 0x7f51c40f51a0 2024-02-19T22:17:58.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:58.234+0000 7f51ca0a6700 1 -- 172.21.15.88:0/3118508021 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51c4105bc0 con 0x7f51c40f51a0 2024-02-19T22:17:58.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:58.234+0000 7f51b9ffb700 1 -- 172.21.15.88:0/3118508021 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f51b4014e20 con 0x7f51c40f51a0 2024-02-19T22:17:58.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:58.235+0000 7f51ca0a6700 1 -- 172.21.15.88:0/3118508021 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51c4106050 con 0x7f51c40f51a0 2024-02-19T22:17:58.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:58.235+0000 7f51ca0a6700 1 -- 172.21.15.88:0/3118508021 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51c4045ad0 con 0x7f51c40f51a0 2024-02-19T22:17:58.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:58.235+0000 7f51b9ffb700 1 -- 172.21.15.88:0/3118508021 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f51b40095f0 con 0x7f51c40f51a0 2024-02-19T22:17:58.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:58.236+0000 7f51b9ffb700 1 --2- 172.21.15.88:0/3118508021 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f51b002ff60 0x7f51b0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:58.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:58.236+0000 7f51b9ffb700 1 -- 172.21.15.88:0/3118508021 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f51b4042cc0 con 0x7f51c40f51a0 2024-02-19T22:17:58.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:58.236+0000 7f51c88a3700 1 --2- 172.21.15.88:0/3118508021 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f51b002ff60 0x7f51b0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:58.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:58.236+0000 7f51c88a3700 1 --2- 172.21.15.88:0/3118508021 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f51b002ff60 0x7f51b0032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f51c00096a0 tx=0x7f51c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:58.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:58.240+0000 7f51b9ffb700 1 -- 172.21.15.88:0/3118508021 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f51b4014420 con 0x7f51c40f51a0 2024-02-19T22:17:58.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:58.442+0000 7f51ca0a6700 1 -- 172.21.15.88:0/3118508021 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f51c4106f00 con 0x7f51c40f51a0 2024-02-19T22:17:58.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:58.443+0000 7f51b9ffb700 1 -- 172.21.15.88:0/3118508021 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f51b4011290 con 0x7f51c40f51a0 2024-02-19T22:17:58.445 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:17:58.445 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:58.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:58.445+0000 7f51ca0a6700 1 -- 172.21.15.88:0/3118508021 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f51b002ff60 msgr2=0x7f51b0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:58.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:58.445+0000 7f51ca0a6700 1 --2- 172.21.15.88:0/3118508021 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f51b002ff60 0x7f51b0032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f51c00096a0 tx=0x7f51c0006b40).stop 2024-02-19T22:17:58.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:58.445+0000 7f51ca0a6700 1 -- 172.21.15.88:0/3118508021 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f51c40f51a0 msgr2=0x7f51c41054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:58.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:58.445+0000 7f51ca0a6700 1 --2- 172.21.15.88:0/3118508021 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f51c40f51a0 0x7f51c41054d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f51b40089f0 tx=0x7f51b400b040).stop 2024-02-19T22:17:58.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:58.445+0000 7f51ca0a6700 1 -- 172.21.15.88:0/3118508021 shutdown_connections 2024-02-19T22:17:58.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:58.445+0000 7f51ca0a6700 1 --2- 172.21.15.88:0/3118508021 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f51b002ff60 0x7f51b0032420 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:58.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:58.445+0000 7f51ca0a6700 1 --2- 172.21.15.88:0/3118508021 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f51c40f51a0 0x7f51c41054d0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:58.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:58.445+0000 7f51ca0a6700 1 -- 172.21.15.88:0/3118508021 >> 172.21.15.88:0/3118508021 conn(0x7f51c40f09a0 msgr2=0x7f51c40f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:58.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:58.445+0000 7f51ca0a6700 1 -- 172.21.15.88:0/3118508021 shutdown_connections 2024-02-19T22:17:58.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:17:58.445+0000 7f51ca0a6700 1 -- 172.21.15.88:0/3118508021 wait complete. 2024-02-19T22:17:58.447 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:17:59.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:59 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:17:57.695363+0000 mgr.smithi052.ghqrtk (mgr.14162) 255 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:59.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:17:59 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:17:58.443720+0000 mon.smithi052 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.88:0/3118508021' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:00.143 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:00.144 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:01.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:01 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:17:59.695704+0000 mgr.smithi052.ghqrtk (mgr.14162) 256 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:02.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:02.137+0000 7f4095009700 1 -- 172.21.15.88:0/2761360131 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f40900f6510 msgr2=0x7f40900f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:02.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:02.137+0000 7f4095009700 1 --2- 172.21.15.88:0/2761360131 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f40900f6510 0x7f40900f6930 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f4084009fb0 tx=0x7f4084009b30).stop 2024-02-19T22:18:02.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:02.137+0000 7f4095009700 1 -- 172.21.15.88:0/2761360131 shutdown_connections 2024-02-19T22:18:02.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:02.137+0000 7f4095009700 1 --2- 172.21.15.88:0/2761360131 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f40900f6510 0x7f40900f6930 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:02.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:02.137+0000 7f4095009700 1 -- 172.21.15.88:0/2761360131 >> 172.21.15.88:0/2761360131 conn(0x7f40900f1c70 msgr2=0x7f40900f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:02.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:02.137+0000 7f4095009700 1 -- 172.21.15.88:0/2761360131 shutdown_connections 2024-02-19T22:18:02.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:02.137+0000 7f4095009700 1 -- 172.21.15.88:0/2761360131 wait complete. 2024-02-19T22:18:02.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:02.137+0000 7f4095009700 1 Processor -- start 2024-02-19T22:18:02.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:02.137+0000 7f4095009700 1 -- start start 2024-02-19T22:18:02.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:02.138+0000 7f4095009700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f40900f6510 0x7f40901067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:02.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:02.138+0000 7f4095009700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4090106cd0 con 0x7f40900f6510 2024-02-19T22:18:02.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:02.138+0000 7f408ed9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f40900f6510 0x7f40901067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:02.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:02.138+0000 7f408ed9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f40900f6510 0x7f40901067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:39706/0 (socket says 172.21.15.88:39706) 2024-02-19T22:18:02.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:02.138+0000 7f408ed9d700 1 -- 172.21.15.88:0/777188830 learned_addr learned my addr 172.21.15.88:0/777188830 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:18:02.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:02.138+0000 7f408ed9d700 1 -- 172.21.15.88:0/777188830 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4084005040 con 0x7f40900f6510 2024-02-19T22:18:02.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:02.138+0000 7f408ed9d700 1 --2- 172.21.15.88:0/777188830 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f40900f6510 0x7f40901067e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f4084005010 tx=0x7f408400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:02.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:02.138+0000 7f40777fe700 1 -- 172.21.15.88:0/777188830 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4084009320 con 0x7f40900f6510 2024-02-19T22:18:02.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:02.139+0000 7f4095009700 1 -- 172.21.15.88:0/777188830 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4090106ed0 con 0x7f40900f6510 2024-02-19T22:18:02.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:02.139+0000 7f40777fe700 1 -- 172.21.15.88:0/777188830 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4084013380 con 0x7f40900f6510 2024-02-19T22:18:02.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:02.139+0000 7f40777fe700 1 -- 172.21.15.88:0/777188830 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4084014e50 con 0x7f40900f6510 2024-02-19T22:18:02.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:02.139+0000 7f4095009700 1 -- 172.21.15.88:0/777188830 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4090107360 con 0x7f40900f6510 2024-02-19T22:18:02.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:02.139+0000 7f4095009700 1 -- 172.21.15.88:0/777188830 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4090045ad0 con 0x7f40900f6510 2024-02-19T22:18:02.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:02.139+0000 7f40777fe700 1 -- 172.21.15.88:0/777188830 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f40840095f0 con 0x7f40900f6510 2024-02-19T22:18:02.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:02.140+0000 7f40777fe700 1 --2- 172.21.15.88:0/777188830 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f407802ff10 0x7f40780323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:02.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:02.140+0000 7f40777fe700 1 -- 172.21.15.88:0/777188830 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4084040e70 con 0x7f40900f6510 2024-02-19T22:18:02.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:02.140+0000 7f408e59c700 1 --2- 172.21.15.88:0/777188830 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f407802ff10 0x7f40780323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:02.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:02.140+0000 7f408e59c700 1 --2- 172.21.15.88:0/777188830 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f407802ff10 0x7f40780323d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f40800096a0 tx=0x7f4080006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:02.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:02.144+0000 7f40777fe700 1 -- 172.21.15.88:0/777188830 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4084014420 con 0x7f40900f6510 2024-02-19T22:18:02.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:02.353+0000 7f4095009700 1 -- 172.21.15.88:0/777188830 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f40901081b0 con 0x7f40900f6510 2024-02-19T22:18:02.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:02.353+0000 7f40777fe700 1 -- 172.21.15.88:0/777188830 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4084010230 con 0x7f40900f6510 2024-02-19T22:18:02.356 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:18:02.356 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:02.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:02.355+0000 7f4095009700 1 -- 172.21.15.88:0/777188830 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f407802ff10 msgr2=0x7f40780323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:02.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:02.355+0000 7f4095009700 1 --2- 172.21.15.88:0/777188830 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f407802ff10 0x7f40780323d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f40800096a0 tx=0x7f4080006b40).stop 2024-02-19T22:18:02.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:02.355+0000 7f4095009700 1 -- 172.21.15.88:0/777188830 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f40900f6510 msgr2=0x7f40901067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:02.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:02.355+0000 7f4095009700 1 --2- 172.21.15.88:0/777188830 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f40900f6510 0x7f40901067e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f4084005010 tx=0x7f408400b040).stop 2024-02-19T22:18:02.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:02.355+0000 7f4095009700 1 -- 172.21.15.88:0/777188830 shutdown_connections 2024-02-19T22:18:02.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:02.355+0000 7f4095009700 1 --2- 172.21.15.88:0/777188830 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f407802ff10 0x7f40780323d0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:02.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:02.355+0000 7f4095009700 1 --2- 172.21.15.88:0/777188830 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f40900f6510 0x7f40901067e0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:02.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:02.355+0000 7f4095009700 1 -- 172.21.15.88:0/777188830 >> 172.21.15.88:0/777188830 conn(0x7f40900f1c70 msgr2=0x7f40900f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:02.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:02.355+0000 7f4095009700 1 -- 172.21.15.88:0/777188830 shutdown_connections 2024-02-19T22:18:02.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:02.355+0000 7f4095009700 1 -- 172.21.15.88:0/777188830 wait complete. 2024-02-19T22:18:02.358 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:18:03.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:03 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:18:01.696130+0000 mgr.smithi052.ghqrtk (mgr.14162) 257 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:03.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:03 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:18:02.354732+0000 mon.smithi052 (mon.0) 266 : audit [DBG] from='client.? 172.21.15.88:0/777188830' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:04.023 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:04.023 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:05.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:05 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:18:03.696521+0000 mgr.smithi052.ghqrtk (mgr.14162) 258 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:05.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:05.856+0000 7fd12ee95700 1 -- 172.21.15.88:0/2646517643 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd1280f5410 msgr2=0x7fd1280f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:05.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:05.856+0000 7fd12ee95700 1 --2- 172.21.15.88:0/2646517643 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd1280f5410 0x7fd1280f5830 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7fd11c009fb0 tx=0x7fd11c009b30).stop 2024-02-19T22:18:05.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:05.856+0000 7fd12ee95700 1 -- 172.21.15.88:0/2646517643 shutdown_connections 2024-02-19T22:18:05.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:05.856+0000 7fd12ee95700 1 --2- 172.21.15.88:0/2646517643 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd1280f5410 0x7fd1280f5830 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:05.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:05.856+0000 7fd12ee95700 1 -- 172.21.15.88:0/2646517643 >> 172.21.15.88:0/2646517643 conn(0x7fd1280f0b70 msgr2=0x7fd1280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:05.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:05.856+0000 7fd12ee95700 1 -- 172.21.15.88:0/2646517643 shutdown_connections 2024-02-19T22:18:05.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:05.856+0000 7fd12ee95700 1 -- 172.21.15.88:0/2646517643 wait complete. 2024-02-19T22:18:05.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:05.857+0000 7fd12ee95700 1 Processor -- start 2024-02-19T22:18:05.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:05.857+0000 7fd12ee95700 1 -- start start 2024-02-19T22:18:05.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:05.857+0000 7fd12ee95700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd1280f5410 0x7fd1281056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:05.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:05.857+0000 7fd12ee95700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd128105bd0 con 0x7fd1280f5410 2024-02-19T22:18:05.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:05.857+0000 7fd12cc31700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd1280f5410 0x7fd1281056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:05.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:05.857+0000 7fd12cc31700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd1280f5410 0x7fd1281056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:35964/0 (socket says 172.21.15.88:35964) 2024-02-19T22:18:05.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:05.857+0000 7fd12cc31700 1 -- 172.21.15.88:0/3817800091 learned_addr learned my addr 172.21.15.88:0/3817800091 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:18:05.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:05.858+0000 7fd12cc31700 1 -- 172.21.15.88:0/3817800091 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd11c005040 con 0x7fd1280f5410 2024-02-19T22:18:05.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:05.858+0000 7fd12cc31700 1 --2- 172.21.15.88:0/3817800091 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd1280f5410 0x7fd1281056e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7fd11c005010 tx=0x7fd11c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:05.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:05.858+0000 7fd1257fa700 1 -- 172.21.15.88:0/3817800091 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd11c009320 con 0x7fd1280f5410 2024-02-19T22:18:05.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:05.858+0000 7fd12ee95700 1 -- 172.21.15.88:0/3817800091 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd128105dd0 con 0x7fd1280f5410 2024-02-19T22:18:05.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:05.858+0000 7fd1257fa700 1 -- 172.21.15.88:0/3817800091 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd11c013380 con 0x7fd1280f5410 2024-02-19T22:18:05.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:05.858+0000 7fd1257fa700 1 -- 172.21.15.88:0/3817800091 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd11c014e50 con 0x7fd1280f5410 2024-02-19T22:18:05.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:05.858+0000 7fd12ee95700 1 -- 172.21.15.88:0/3817800091 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd128106260 con 0x7fd1280f5410 2024-02-19T22:18:05.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:05.859+0000 7fd1257fa700 1 -- 172.21.15.88:0/3817800091 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd11c0095f0 con 0x7fd1280f5410 2024-02-19T22:18:05.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:05.859+0000 7fd12ee95700 1 -- 172.21.15.88:0/3817800091 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd128045ad0 con 0x7fd1280f5410 2024-02-19T22:18:05.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:05.859+0000 7fd1257fa700 1 --2- 172.21.15.88:0/3817800091 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd11802ff10 0x7fd1180323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:05.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:05.859+0000 7fd1257fa700 1 -- 172.21.15.88:0/3817800091 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd11c040fa0 con 0x7fd1280f5410 2024-02-19T22:18:05.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:05.859+0000 7fd127fff700 1 --2- 172.21.15.88:0/3817800091 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd11802ff10 0x7fd1180323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:05.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:05.860+0000 7fd127fff700 1 --2- 172.21.15.88:0/3817800091 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd11802ff10 0x7fd1180323d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fd1140096a0 tx=0x7fd114006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:05.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:05.864+0000 7fd1257fa700 1 -- 172.21.15.88:0/3817800091 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd11c014420 con 0x7fd1280f5410 2024-02-19T22:18:06.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:06.071+0000 7fd12ee95700 1 -- 172.21.15.88:0/3817800091 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd128107110 con 0x7fd1280f5410 2024-02-19T22:18:06.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:06.072+0000 7fd1257fa700 1 -- 172.21.15.88:0/3817800091 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd11c010230 con 0x7fd1280f5410 2024-02-19T22:18:06.074 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:18:06.074 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:06.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:06.073+0000 7fd12ee95700 1 -- 172.21.15.88:0/3817800091 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd11802ff10 msgr2=0x7fd1180323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:06.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:06.073+0000 7fd12ee95700 1 --2- 172.21.15.88:0/3817800091 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd11802ff10 0x7fd1180323d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fd1140096a0 tx=0x7fd114006b40).stop 2024-02-19T22:18:06.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:06.073+0000 7fd12ee95700 1 -- 172.21.15.88:0/3817800091 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd1280f5410 msgr2=0x7fd1281056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:06.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:06.073+0000 7fd12ee95700 1 --2- 172.21.15.88:0/3817800091 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd1280f5410 0x7fd1281056e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7fd11c005010 tx=0x7fd11c00b040).stop 2024-02-19T22:18:06.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:06.073+0000 7fd12ee95700 1 -- 172.21.15.88:0/3817800091 shutdown_connections 2024-02-19T22:18:06.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:06.073+0000 7fd12ee95700 1 --2- 172.21.15.88:0/3817800091 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd11802ff10 0x7fd1180323d0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:06.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:06.073+0000 7fd12ee95700 1 --2- 172.21.15.88:0/3817800091 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd1280f5410 0x7fd1281056e0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:06.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:06.073+0000 7fd12ee95700 1 -- 172.21.15.88:0/3817800091 >> 172.21.15.88:0/3817800091 conn(0x7fd1280f0b70 msgr2=0x7fd1280f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:06.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:06.073+0000 7fd12ee95700 1 -- 172.21.15.88:0/3817800091 shutdown_connections 2024-02-19T22:18:06.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:06.073+0000 7fd12ee95700 1 -- 172.21.15.88:0/3817800091 wait complete. 2024-02-19T22:18:06.077 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:18:06.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:06 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:18:06.072831+0000 mon.smithi052 (mon.0) 267 : audit [DBG] from='client.? 172.21.15.88:0/3817800091' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:07.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:07 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:18:05.696815+0000 mgr.smithi052.ghqrtk (mgr.14162) 259 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:07.801 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:07.802 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:09.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:09 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:18:07.697182+0000 mgr.smithi052.ghqrtk (mgr.14162) 260 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:09.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:09.596+0000 7f9697b6d700 1 -- 172.21.15.88:0/62647614 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f96900f5410 msgr2=0x7f96900f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:09.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:09.596+0000 7f9697b6d700 1 --2- 172.21.15.88:0/62647614 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f96900f5410 0x7f96900f5830 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f9680009fb0 tx=0x7f9680009b30).stop 2024-02-19T22:18:09.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:09.596+0000 7f9697b6d700 1 -- 172.21.15.88:0/62647614 shutdown_connections 2024-02-19T22:18:09.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:09.596+0000 7f9697b6d700 1 --2- 172.21.15.88:0/62647614 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f96900f5410 0x7f96900f5830 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:09.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:09.596+0000 7f9697b6d700 1 -- 172.21.15.88:0/62647614 >> 172.21.15.88:0/62647614 conn(0x7f96900f0b70 msgr2=0x7f96900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:09.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:09.597+0000 7f9697b6d700 1 -- 172.21.15.88:0/62647614 shutdown_connections 2024-02-19T22:18:09.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:09.597+0000 7f9697b6d700 1 -- 172.21.15.88:0/62647614 wait complete. 2024-02-19T22:18:09.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:09.597+0000 7f9697b6d700 1 Processor -- start 2024-02-19T22:18:09.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:09.597+0000 7f9697b6d700 1 -- start start 2024-02-19T22:18:09.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:09.597+0000 7f9697b6d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f96900f5410 0x7f96901089f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:09.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:09.597+0000 7f9697b6d700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9690108ee0 con 0x7f96900f5410 2024-02-19T22:18:09.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:09.597+0000 7f9695909700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f96900f5410 0x7f96901089f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:09.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:09.597+0000 7f9695909700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f96900f5410 0x7f96901089f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:35986/0 (socket says 172.21.15.88:35986) 2024-02-19T22:18:09.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:09.597+0000 7f9695909700 1 -- 172.21.15.88:0/1681895166 learned_addr learned my addr 172.21.15.88:0/1681895166 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:18:09.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:09.598+0000 7f9695909700 1 -- 172.21.15.88:0/1681895166 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9680005040 con 0x7f96900f5410 2024-02-19T22:18:09.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:09.598+0000 7f9695909700 1 --2- 172.21.15.88:0/1681895166 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f96900f5410 0x7f96901089f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f9680005010 tx=0x7f968000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:09.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:09.598+0000 7f96867fc700 1 -- 172.21.15.88:0/1681895166 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9680009320 con 0x7f96900f5410 2024-02-19T22:18:09.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:09.598+0000 7f96867fc700 1 -- 172.21.15.88:0/1681895166 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9680013380 con 0x7f96900f5410 2024-02-19T22:18:09.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:09.598+0000 7f9697b6d700 1 -- 172.21.15.88:0/1681895166 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96901090e0 con 0x7f96900f5410 2024-02-19T22:18:09.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:09.598+0000 7f96867fc700 1 -- 172.21.15.88:0/1681895166 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9680014e50 con 0x7f96900f5410 2024-02-19T22:18:09.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:09.598+0000 7f9697b6d700 1 -- 172.21.15.88:0/1681895166 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9690109570 con 0x7f96900f5410 2024-02-19T22:18:09.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:09.599+0000 7f96867fc700 1 -- 172.21.15.88:0/1681895166 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9680014420 con 0x7f96900f5410 2024-02-19T22:18:09.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:09.599+0000 7f9697b6d700 1 -- 172.21.15.88:0/1681895166 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9690045ad0 con 0x7f96900f5410 2024-02-19T22:18:09.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:09.599+0000 7f96867fc700 1 --2- 172.21.15.88:0/1681895166 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f967c02ff10 0x7f967c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:09.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:09.599+0000 7f96867fc700 1 -- 172.21.15.88:0/1681895166 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9680040f40 con 0x7f96900f5410 2024-02-19T22:18:09.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:09.599+0000 7f9695108700 1 --2- 172.21.15.88:0/1681895166 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f967c02ff10 0x7f967c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:09.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:09.600+0000 7f9695108700 1 --2- 172.21.15.88:0/1681895166 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f967c02ff10 0x7f967c0323d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f968c0096a0 tx=0x7f968c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:09.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:09.604+0000 7f96867fc700 1 -- 172.21.15.88:0/1681895166 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f96800095f0 con 0x7f96900f5410 2024-02-19T22:18:09.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:09.807+0000 7f9697b6d700 1 -- 172.21.15.88:0/1681895166 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f96900fafa0 con 0x7f96900f5410 2024-02-19T22:18:09.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:09.808+0000 7f96867fc700 1 -- 172.21.15.88:0/1681895166 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f968001db70 con 0x7f96900f5410 2024-02-19T22:18:09.810 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:18:09.810 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:09.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:09.809+0000 7f9697b6d700 1 -- 172.21.15.88:0/1681895166 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f967c02ff10 msgr2=0x7f967c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:09.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:09.809+0000 7f9697b6d700 1 --2- 172.21.15.88:0/1681895166 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f967c02ff10 0x7f967c0323d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f968c0096a0 tx=0x7f968c006b40).stop 2024-02-19T22:18:09.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:09.809+0000 7f9697b6d700 1 -- 172.21.15.88:0/1681895166 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f96900f5410 msgr2=0x7f96901089f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:09.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:09.809+0000 7f9697b6d700 1 --2- 172.21.15.88:0/1681895166 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f96900f5410 0x7f96901089f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f9680005010 tx=0x7f968000b040).stop 2024-02-19T22:18:09.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:09.810+0000 7f9697b6d700 1 -- 172.21.15.88:0/1681895166 shutdown_connections 2024-02-19T22:18:09.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:09.810+0000 7f9697b6d700 1 --2- 172.21.15.88:0/1681895166 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f967c02ff10 0x7f967c0323d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:09.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:09.810+0000 7f9697b6d700 1 --2- 172.21.15.88:0/1681895166 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f96900f5410 0x7f96901089f0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:09.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:09.810+0000 7f9697b6d700 1 -- 172.21.15.88:0/1681895166 >> 172.21.15.88:0/1681895166 conn(0x7f96900f0b70 msgr2=0x7f969005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:09.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:09.810+0000 7f9697b6d700 1 -- 172.21.15.88:0/1681895166 shutdown_connections 2024-02-19T22:18:09.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:09.810+0000 7f9697b6d700 1 -- 172.21.15.88:0/1681895166 wait complete. 2024-02-19T22:18:09.813 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:18:10.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:10 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:18:09.809379+0000 mon.smithi052 (mon.0) 268 : audit [DBG] from='client.? 172.21.15.88:0/1681895166' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:11.555 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:11.555 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:11.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:11 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:18:09.697561+0000 mgr.smithi052.ghqrtk (mgr.14162) 261 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:13.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:13.337+0000 7f2db3515700 1 -- 172.21.15.88:0/1351410317 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2dac060220 msgr2=0x7f2dac0f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:13.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:13.337+0000 7f2db3515700 1 --2- 172.21.15.88:0/1351410317 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2dac060220 0x7f2dac0f9150 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f2da0009fb0 tx=0x7f2da0009b30).stop 2024-02-19T22:18:13.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:13.337+0000 7f2db3515700 1 -- 172.21.15.88:0/1351410317 shutdown_connections 2024-02-19T22:18:13.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:13.337+0000 7f2db3515700 1 --2- 172.21.15.88:0/1351410317 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2dac060220 0x7f2dac0f9150 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:13.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:13.337+0000 7f2db3515700 1 -- 172.21.15.88:0/1351410317 >> 172.21.15.88:0/1351410317 conn(0x7f2dac0f0b70 msgr2=0x7f2dac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:13.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:13.337+0000 7f2db3515700 1 -- 172.21.15.88:0/1351410317 shutdown_connections 2024-02-19T22:18:13.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:13.337+0000 7f2db3515700 1 -- 172.21.15.88:0/1351410317 wait complete. 2024-02-19T22:18:13.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:13.337+0000 7f2db3515700 1 Processor -- start 2024-02-19T22:18:13.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:13.337+0000 7f2db3515700 1 -- start start 2024-02-19T22:18:13.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:13.337+0000 7f2db3515700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2dac060220 0x7f2dac1067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:13.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:13.337+0000 7f2db3515700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dac106c90 con 0x7f2dac060220 2024-02-19T22:18:13.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:13.338+0000 7f2db12b1700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2dac060220 0x7f2dac1067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:13.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:13.338+0000 7f2db12b1700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2dac060220 0x7f2dac1067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:40100/0 (socket says 172.21.15.88:40100) 2024-02-19T22:18:13.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:13.338+0000 7f2db12b1700 1 -- 172.21.15.88:0/1037806578 learned_addr learned my addr 172.21.15.88:0/1037806578 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:18:13.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:13.338+0000 7f2db12b1700 1 -- 172.21.15.88:0/1037806578 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2da0005040 con 0x7f2dac060220 2024-02-19T22:18:13.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:13.338+0000 7f2db12b1700 1 --2- 172.21.15.88:0/1037806578 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2dac060220 0x7f2dac1067a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f2da00089f0 tx=0x7f2da000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:13.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:13.338+0000 7f2d9dffb700 1 -- 172.21.15.88:0/1037806578 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2da0009320 con 0x7f2dac060220 2024-02-19T22:18:13.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:13.338+0000 7f2db3515700 1 -- 172.21.15.88:0/1037806578 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dac106e90 con 0x7f2dac060220 2024-02-19T22:18:13.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:13.338+0000 7f2d9dffb700 1 -- 172.21.15.88:0/1037806578 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2da0013380 con 0x7f2dac060220 2024-02-19T22:18:13.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:13.338+0000 7f2d9dffb700 1 -- 172.21.15.88:0/1037806578 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2da0014e20 con 0x7f2dac060220 2024-02-19T22:18:13.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:13.338+0000 7f2db3515700 1 -- 172.21.15.88:0/1037806578 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dac107320 con 0x7f2dac060220 2024-02-19T22:18:13.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:13.339+0000 7f2db3515700 1 -- 172.21.15.88:0/1037806578 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dac045ad0 con 0x7f2dac060220 2024-02-19T22:18:13.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:13.339+0000 7f2d9dffb700 1 -- 172.21.15.88:0/1037806578 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2da00095f0 con 0x7f2dac060220 2024-02-19T22:18:13.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:13.339+0000 7f2d9dffb700 1 --2- 172.21.15.88:0/1037806578 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f2d9802ff10 0x7f2d980323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:13.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:13.339+0000 7f2d9dffb700 1 -- 172.21.15.88:0/1037806578 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2da0040e90 con 0x7f2dac060220 2024-02-19T22:18:13.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:13.340+0000 7f2db0ab0700 1 --2- 172.21.15.88:0/1037806578 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f2d9802ff10 0x7f2d980323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:13.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:13.340+0000 7f2db0ab0700 1 --2- 172.21.15.88:0/1037806578 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f2d9802ff10 0x7f2d980323d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f2da80096a0 tx=0x7f2da8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:13.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:13.344+0000 7f2d9dffb700 1 -- 172.21.15.88:0/1037806578 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2da0014420 con 0x7f2dac060220 2024-02-19T22:18:13.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:13.548+0000 7f2db3515700 1 -- 172.21.15.88:0/1037806578 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2dac1081d0 con 0x7f2dac060220 2024-02-19T22:18:13.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:13.548+0000 7f2d9dffb700 1 -- 172.21.15.88:0/1037806578 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2da001d980 con 0x7f2dac060220 2024-02-19T22:18:13.551 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:18:13.551 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:13.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:13.550+0000 7f2db3515700 1 -- 172.21.15.88:0/1037806578 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f2d9802ff10 msgr2=0x7f2d980323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:13.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:13.550+0000 7f2db3515700 1 --2- 172.21.15.88:0/1037806578 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f2d9802ff10 0x7f2d980323d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f2da80096a0 tx=0x7f2da8006b40).stop 2024-02-19T22:18:13.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:13.550+0000 7f2db3515700 1 -- 172.21.15.88:0/1037806578 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2dac060220 msgr2=0x7f2dac1067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:13.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:13.550+0000 7f2db3515700 1 --2- 172.21.15.88:0/1037806578 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2dac060220 0x7f2dac1067a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f2da00089f0 tx=0x7f2da000b040).stop 2024-02-19T22:18:13.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:13.550+0000 7f2db3515700 1 -- 172.21.15.88:0/1037806578 shutdown_connections 2024-02-19T22:18:13.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:13.550+0000 7f2db3515700 1 --2- 172.21.15.88:0/1037806578 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f2d9802ff10 0x7f2d980323d0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:13.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:13.550+0000 7f2db3515700 1 --2- 172.21.15.88:0/1037806578 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2dac060220 0x7f2dac1067a0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:13.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:13.550+0000 7f2db3515700 1 -- 172.21.15.88:0/1037806578 >> 172.21.15.88:0/1037806578 conn(0x7f2dac0f0b70 msgr2=0x7f2dac0f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:13.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:13.550+0000 7f2db3515700 1 -- 172.21.15.88:0/1037806578 shutdown_connections 2024-02-19T22:18:13.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:13.550+0000 7f2db3515700 1 -- 172.21.15.88:0/1037806578 wait complete. 2024-02-19T22:18:13.553 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:18:13.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:13 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:18:11.697902+0000 mgr.smithi052.ghqrtk (mgr.14162) 262 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:14.595 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:14 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:18:13.549684+0000 mon.smithi052 (mon.0) 269 : audit [DBG] from='client.? 172.21.15.88:0/1037806578' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:15.067 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:15.067 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:15.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:15 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:18:13.698261+0000 mgr.smithi052.ghqrtk (mgr.14162) 263 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:16.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:16.914+0000 7f1112a2a700 1 -- 172.21.15.88:0/1700576547 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f110c0f5410 msgr2=0x7f110c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:16.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:16.914+0000 7f1112a2a700 1 --2- 172.21.15.88:0/1700576547 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f110c0f5410 0x7f110c0f5830 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f10fc009fb0 tx=0x7f10fc009b30).stop 2024-02-19T22:18:16.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:16.914+0000 7f1112a2a700 1 -- 172.21.15.88:0/1700576547 shutdown_connections 2024-02-19T22:18:16.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:16.914+0000 7f1112a2a700 1 --2- 172.21.15.88:0/1700576547 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f110c0f5410 0x7f110c0f5830 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:16.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:16.914+0000 7f1112a2a700 1 -- 172.21.15.88:0/1700576547 >> 172.21.15.88:0/1700576547 conn(0x7f110c0f0b70 msgr2=0x7f110c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:16.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:16.914+0000 7f1112a2a700 1 -- 172.21.15.88:0/1700576547 shutdown_connections 2024-02-19T22:18:16.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:16.914+0000 7f1112a2a700 1 -- 172.21.15.88:0/1700576547 wait complete. 2024-02-19T22:18:16.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:16.915+0000 7f1112a2a700 1 Processor -- start 2024-02-19T22:18:16.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:16.915+0000 7f1112a2a700 1 -- start start 2024-02-19T22:18:16.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:16.915+0000 7f1112a2a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f110c0f5410 0x7f110c0fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:16.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:16.915+0000 7f1112a2a700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f110c0fb960 con 0x7f110c0f5410 2024-02-19T22:18:16.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:16.915+0000 7f110bfff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f110c0f5410 0x7f110c0fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:16.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:16.915+0000 7f110bfff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f110c0f5410 0x7f110c0fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:40116/0 (socket says 172.21.15.88:40116) 2024-02-19T22:18:16.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:16.915+0000 7f110bfff700 1 -- 172.21.15.88:0/2961749833 learned_addr learned my addr 172.21.15.88:0/2961749833 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:18:16.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:16.916+0000 7f110bfff700 1 -- 172.21.15.88:0/2961749833 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10fc005040 con 0x7f110c0f5410 2024-02-19T22:18:16.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:16.916+0000 7f110bfff700 1 --2- 172.21.15.88:0/2961749833 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f110c0f5410 0x7f110c0fd4e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f10fc005010 tx=0x7f10fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:16.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:16.916+0000 7f1108ff9700 1 -- 172.21.15.88:0/2961749833 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10fc009320 con 0x7f110c0f5410 2024-02-19T22:18:16.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:16.916+0000 7f1112a2a700 1 -- 172.21.15.88:0/2961749833 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f110c0fbb00 con 0x7f110c0f5410 2024-02-19T22:18:16.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:16.916+0000 7f1108ff9700 1 -- 172.21.15.88:0/2961749833 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f10fc013380 con 0x7f110c0f5410 2024-02-19T22:18:16.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:16.916+0000 7f1108ff9700 1 -- 172.21.15.88:0/2961749833 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10fc014e50 con 0x7f110c0f5410 2024-02-19T22:18:16.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:16.916+0000 7f1112a2a700 1 -- 172.21.15.88:0/2961749833 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f110c0fbf90 con 0x7f110c0f5410 2024-02-19T22:18:16.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:16.917+0000 7f1108ff9700 1 -- 172.21.15.88:0/2961749833 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f10fc014420 con 0x7f110c0f5410 2024-02-19T22:18:16.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:16.917+0000 7f1112a2a700 1 -- 172.21.15.88:0/2961749833 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f110c059100 con 0x7f110c0f5410 2024-02-19T22:18:16.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:16.917+0000 7f1108ff9700 1 --2- 172.21.15.88:0/2961749833 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f10f402ff10 0x7f10f40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:16.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:16.917+0000 7f1108ff9700 1 -- 172.21.15.88:0/2961749833 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f10fc040f40 con 0x7f110c0f5410 2024-02-19T22:18:16.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:16.917+0000 7f110b7fe700 1 --2- 172.21.15.88:0/2961749833 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f10f402ff10 0x7f10f40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:16.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:16.918+0000 7f110b7fe700 1 --2- 172.21.15.88:0/2961749833 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f10f402ff10 0x7f10f40323d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f11000096a0 tx=0x7f1100006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:16.924 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:16.922+0000 7f1108ff9700 1 -- 172.21.15.88:0/2961749833 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f10fc0095f0 con 0x7f110c0f5410 2024-02-19T22:18:17.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:17.129+0000 7f1112a2a700 1 -- 172.21.15.88:0/2961749833 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f110c0fafa0 con 0x7f110c0f5410 2024-02-19T22:18:17.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:17.130+0000 7f1108ff9700 1 -- 172.21.15.88:0/2961749833 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f10fc01db70 con 0x7f110c0f5410 2024-02-19T22:18:17.132 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:18:17.132 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:17.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:17.131+0000 7f1112a2a700 1 -- 172.21.15.88:0/2961749833 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f10f402ff10 msgr2=0x7f10f40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:17.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:17.131+0000 7f1112a2a700 1 --2- 172.21.15.88:0/2961749833 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f10f402ff10 0x7f10f40323d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f11000096a0 tx=0x7f1100006b40).stop 2024-02-19T22:18:17.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:17.131+0000 7f1112a2a700 1 -- 172.21.15.88:0/2961749833 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f110c0f5410 msgr2=0x7f110c0fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:17.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:17.131+0000 7f1112a2a700 1 --2- 172.21.15.88:0/2961749833 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f110c0f5410 0x7f110c0fd4e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f10fc005010 tx=0x7f10fc00b040).stop 2024-02-19T22:18:17.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:17.131+0000 7f1112a2a700 1 -- 172.21.15.88:0/2961749833 shutdown_connections 2024-02-19T22:18:17.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:17.131+0000 7f1112a2a700 1 --2- 172.21.15.88:0/2961749833 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f10f402ff10 0x7f10f40323d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:17.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:17.131+0000 7f1112a2a700 1 --2- 172.21.15.88:0/2961749833 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f110c0f5410 0x7f110c0fd4e0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:17.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:17.131+0000 7f1112a2a700 1 -- 172.21.15.88:0/2961749833 >> 172.21.15.88:0/2961749833 conn(0x7f110c0f0b70 msgr2=0x7f110c05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:17.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:17.131+0000 7f1112a2a700 1 -- 172.21.15.88:0/2961749833 shutdown_connections 2024-02-19T22:18:17.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:17.131+0000 7f1112a2a700 1 -- 172.21.15.88:0/2961749833 wait complete. 2024-02-19T22:18:17.135 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:18:17.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:17 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:18:15.698589+0000 mgr.smithi052.ghqrtk (mgr.14162) 264 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:17.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:17 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:18:17.131113+0000 mon.smithi052 (mon.0) 270 : audit [DBG] from='client.? 172.21.15.88:0/2961749833' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:18.869 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:18.870 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:19.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:19 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:18:17.698909+0000 mgr.smithi052.ghqrtk (mgr.14162) 265 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:20.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:20.760+0000 7f15debd8700 1 -- 172.21.15.88:0/3917677823 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f15d80f5410 msgr2=0x7f15d80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:20.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:20.760+0000 7f15debd8700 1 --2- 172.21.15.88:0/3917677823 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f15d80f5410 0x7f15d80f5830 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f15cc009fb0 tx=0x7f15cc009b30).stop 2024-02-19T22:18:20.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:20.760+0000 7f15debd8700 1 -- 172.21.15.88:0/3917677823 shutdown_connections 2024-02-19T22:18:20.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:20.760+0000 7f15debd8700 1 --2- 172.21.15.88:0/3917677823 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f15d80f5410 0x7f15d80f5830 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:20.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:20.760+0000 7f15debd8700 1 -- 172.21.15.88:0/3917677823 >> 172.21.15.88:0/3917677823 conn(0x7f15d80f0b70 msgr2=0x7f15d80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:20.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:20.760+0000 7f15debd8700 1 -- 172.21.15.88:0/3917677823 shutdown_connections 2024-02-19T22:18:20.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:20.760+0000 7f15debd8700 1 -- 172.21.15.88:0/3917677823 wait complete. 2024-02-19T22:18:20.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:20.760+0000 7f15debd8700 1 Processor -- start 2024-02-19T22:18:20.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:20.760+0000 7f15debd8700 1 -- start start 2024-02-19T22:18:20.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:20.761+0000 7f15debd8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f15d80f5410 0x7f15d81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:20.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:20.761+0000 7f15debd8700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d8106cd0 con 0x7f15d80f5410 2024-02-19T22:18:20.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:20.761+0000 7f15dc974700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f15d80f5410 0x7f15d81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:20.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:20.761+0000 7f15dc974700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f15d80f5410 0x7f15d81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:40130/0 (socket says 172.21.15.88:40130) 2024-02-19T22:18:20.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:20.761+0000 7f15dc974700 1 -- 172.21.15.88:0/1367920226 learned_addr learned my addr 172.21.15.88:0/1367920226 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:18:20.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:20.761+0000 7f15dc974700 1 -- 172.21.15.88:0/1367920226 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15cc005040 con 0x7f15d80f5410 2024-02-19T22:18:20.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:20.762+0000 7f15dc974700 1 --2- 172.21.15.88:0/1367920226 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f15d80f5410 0x7f15d81067e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f15cc005010 tx=0x7f15cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:20.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:20.762+0000 7f15d57fa700 1 -- 172.21.15.88:0/1367920226 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f15cc009320 con 0x7f15d80f5410 2024-02-19T22:18:20.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:20.762+0000 7f15debd8700 1 -- 172.21.15.88:0/1367920226 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15d8106ed0 con 0x7f15d80f5410 2024-02-19T22:18:20.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:20.762+0000 7f15d57fa700 1 -- 172.21.15.88:0/1367920226 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f15cc013380 con 0x7f15d80f5410 2024-02-19T22:18:20.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:20.762+0000 7f15d57fa700 1 -- 172.21.15.88:0/1367920226 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f15cc014e50 con 0x7f15d80f5410 2024-02-19T22:18:20.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:20.762+0000 7f15debd8700 1 -- 172.21.15.88:0/1367920226 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15d8107360 con 0x7f15d80f5410 2024-02-19T22:18:20.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:20.763+0000 7f15d57fa700 1 -- 172.21.15.88:0/1367920226 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f15cc014420 con 0x7f15d80f5410 2024-02-19T22:18:20.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:20.763+0000 7f15debd8700 1 -- 172.21.15.88:0/1367920226 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15d8045ad0 con 0x7f15d80f5410 2024-02-19T22:18:20.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:20.763+0000 7f15d57fa700 1 --2- 172.21.15.88:0/1367920226 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f15c802ff10 0x7f15c80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:20.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:20.763+0000 7f15d57fa700 1 -- 172.21.15.88:0/1367920226 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f15cc040ee0 con 0x7f15d80f5410 2024-02-19T22:18:20.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:20.763+0000 7f15d7fff700 1 --2- 172.21.15.88:0/1367920226 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f15c802ff10 0x7f15c80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:20.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:20.763+0000 7f15d7fff700 1 --2- 172.21.15.88:0/1367920226 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f15c802ff10 0x7f15c80323d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f15c40096a0 tx=0x7f15c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:20.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:20.768+0000 7f15d57fa700 1 -- 172.21.15.88:0/1367920226 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f15cc0095f0 con 0x7f15d80f5410 2024-02-19T22:18:20.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:20.971+0000 7f15debd8700 1 -- 172.21.15.88:0/1367920226 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f15d80fafa0 con 0x7f15d80f5410 2024-02-19T22:18:20.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:20.972+0000 7f15d57fa700 1 -- 172.21.15.88:0/1367920226 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f15cc01db70 con 0x7f15d80f5410 2024-02-19T22:18:20.974 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:18:20.974 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:20.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:20.973+0000 7f15debd8700 1 -- 172.21.15.88:0/1367920226 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f15c802ff10 msgr2=0x7f15c80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:20.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:20.973+0000 7f15debd8700 1 --2- 172.21.15.88:0/1367920226 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f15c802ff10 0x7f15c80323d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f15c40096a0 tx=0x7f15c4006b40).stop 2024-02-19T22:18:20.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:20.973+0000 7f15debd8700 1 -- 172.21.15.88:0/1367920226 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f15d80f5410 msgr2=0x7f15d81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:20.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:20.973+0000 7f15debd8700 1 --2- 172.21.15.88:0/1367920226 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f15d80f5410 0x7f15d81067e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f15cc005010 tx=0x7f15cc00b040).stop 2024-02-19T22:18:20.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:20.973+0000 7f15debd8700 1 -- 172.21.15.88:0/1367920226 shutdown_connections 2024-02-19T22:18:20.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:20.973+0000 7f15debd8700 1 --2- 172.21.15.88:0/1367920226 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f15c802ff10 0x7f15c80323d0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:20.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:20.973+0000 7f15debd8700 1 --2- 172.21.15.88:0/1367920226 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f15d80f5410 0x7f15d81067e0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:20.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:20.973+0000 7f15debd8700 1 -- 172.21.15.88:0/1367920226 >> 172.21.15.88:0/1367920226 conn(0x7f15d80f0b70 msgr2=0x7f15d805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:20.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:20.973+0000 7f15debd8700 1 -- 172.21.15.88:0/1367920226 shutdown_connections 2024-02-19T22:18:20.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:20.973+0000 7f15debd8700 1 -- 172.21.15.88:0/1367920226 wait complete. 2024-02-19T22:18:20.977 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:18:21.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:21 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:18:19.699300+0000 mgr.smithi052.ghqrtk (mgr.14162) 266 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:21.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:21 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:18:20.973216+0000 mon.smithi052 (mon.0) 271 : audit [DBG] from='client.? 172.21.15.88:0/1367920226' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:22.699 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:22.700 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:23.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:23 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:18:21.699674+0000 mgr.smithi052.ghqrtk (mgr.14162) 267 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:24.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:24.459+0000 7f5bcead3700 1 -- 172.21.15.88:0/288313810 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5bc80f78e0 msgr2=0x7f5bc80f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:24.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:24.459+0000 7f5bcead3700 1 --2- 172.21.15.88:0/288313810 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5bc80f78e0 0x7f5bc80f7d00 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f5bbc009fb0 tx=0x7f5bbc009b30).stop 2024-02-19T22:18:24.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:24.459+0000 7f5bcead3700 1 -- 172.21.15.88:0/288313810 shutdown_connections 2024-02-19T22:18:24.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:24.459+0000 7f5bcead3700 1 --2- 172.21.15.88:0/288313810 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5bc80f78e0 0x7f5bc80f7d00 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:24.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:24.459+0000 7f5bcead3700 1 -- 172.21.15.88:0/288313810 >> 172.21.15.88:0/288313810 conn(0x7f5bc80f0120 msgr2=0x7f5bc80f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:24.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:24.459+0000 7f5bcead3700 1 -- 172.21.15.88:0/288313810 shutdown_connections 2024-02-19T22:18:24.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:24.459+0000 7f5bcead3700 1 -- 172.21.15.88:0/288313810 wait complete. 2024-02-19T22:18:24.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:24.459+0000 7f5bcead3700 1 Processor -- start 2024-02-19T22:18:24.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:24.459+0000 7f5bcead3700 1 -- start start 2024-02-19T22:18:24.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:24.459+0000 7f5bcead3700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5bc80f78e0 0x7f5bc80f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:24.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:24.459+0000 7f5bcead3700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc80f5630 con 0x7f5bc80f78e0 2024-02-19T22:18:24.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:24.460+0000 7f5bcdad1700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5bc80f78e0 0x7f5bc80f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:24.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:24.460+0000 7f5bcdad1700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5bc80f78e0 0x7f5bc80f71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:50780/0 (socket says 172.21.15.88:50780) 2024-02-19T22:18:24.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:24.460+0000 7f5bcdad1700 1 -- 172.21.15.88:0/3312970860 learned_addr learned my addr 172.21.15.88:0/3312970860 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:18:24.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:24.460+0000 7f5bcdad1700 1 -- 172.21.15.88:0/3312970860 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bbc005040 con 0x7f5bc80f78e0 2024-02-19T22:18:24.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:24.460+0000 7f5bcdad1700 1 --2- 172.21.15.88:0/3312970860 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5bc80f78e0 0x7f5bc80f71b0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f5bbc0089f0 tx=0x7f5bbc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:24.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:24.460+0000 7f5bba7fc700 1 -- 172.21.15.88:0/3312970860 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5bbc009320 con 0x7f5bc80f78e0 2024-02-19T22:18:24.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:24.460+0000 7f5bcead3700 1 -- 172.21.15.88:0/3312970860 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bc80f57d0 con 0x7f5bc80f78e0 2024-02-19T22:18:24.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:24.460+0000 7f5bba7fc700 1 -- 172.21.15.88:0/3312970860 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5bbc013380 con 0x7f5bc80f78e0 2024-02-19T22:18:24.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:24.460+0000 7f5bba7fc700 1 -- 172.21.15.88:0/3312970860 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5bbc014e20 con 0x7f5bc80f78e0 2024-02-19T22:18:24.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:24.460+0000 7f5bcead3700 1 -- 172.21.15.88:0/3312970860 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bc80f5c60 con 0x7f5bc80f78e0 2024-02-19T22:18:24.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:24.461+0000 7f5bba7fc700 1 -- 172.21.15.88:0/3312970860 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5bbc0095f0 con 0x7f5bc80f78e0 2024-02-19T22:18:24.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:24.461+0000 7f5bcead3700 1 -- 172.21.15.88:0/3312970860 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bc8059100 con 0x7f5bc80f78e0 2024-02-19T22:18:24.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:24.461+0000 7f5bba7fc700 1 --2- 172.21.15.88:0/3312970860 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f5bb402ff10 0x7f5bb40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:24.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:24.461+0000 7f5bba7fc700 1 -- 172.21.15.88:0/3312970860 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5bbc040e90 con 0x7f5bc80f78e0 2024-02-19T22:18:24.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:24.462+0000 7f5bcd2d0700 1 --2- 172.21.15.88:0/3312970860 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f5bb402ff10 0x7f5bb40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:24.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:24.462+0000 7f5bcd2d0700 1 --2- 172.21.15.88:0/3312970860 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f5bb402ff10 0x7f5bb40323d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f5bc40096a0 tx=0x7f5bc4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:24.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:24.466+0000 7f5bba7fc700 1 -- 172.21.15.88:0/3312970860 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5bbc014420 con 0x7f5bc80f78e0 2024-02-19T22:18:24.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:24.671+0000 7f5bcead3700 1 -- 172.21.15.88:0/3312970860 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5bc80fbe20 con 0x7f5bc80f78e0 2024-02-19T22:18:24.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:24.671+0000 7f5bba7fc700 1 -- 172.21.15.88:0/3312970860 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5bbc01d980 con 0x7f5bc80f78e0 2024-02-19T22:18:24.676 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:18:24.676 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:24.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:24.673+0000 7f5bcead3700 1 -- 172.21.15.88:0/3312970860 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f5bb402ff10 msgr2=0x7f5bb40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:24.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:24.673+0000 7f5bcead3700 1 --2- 172.21.15.88:0/3312970860 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f5bb402ff10 0x7f5bb40323d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f5bc40096a0 tx=0x7f5bc4006b40).stop 2024-02-19T22:18:24.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:24.673+0000 7f5bcead3700 1 -- 172.21.15.88:0/3312970860 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5bc80f78e0 msgr2=0x7f5bc80f71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:24.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:24.673+0000 7f5bcead3700 1 --2- 172.21.15.88:0/3312970860 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5bc80f78e0 0x7f5bc80f71b0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f5bbc0089f0 tx=0x7f5bbc00b040).stop 2024-02-19T22:18:24.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:24.673+0000 7f5bcead3700 1 -- 172.21.15.88:0/3312970860 shutdown_connections 2024-02-19T22:18:24.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:24.673+0000 7f5bcead3700 1 --2- 172.21.15.88:0/3312970860 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f5bb402ff10 0x7f5bb40323d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:24.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:24.673+0000 7f5bcead3700 1 --2- 172.21.15.88:0/3312970860 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5bc80f78e0 0x7f5bc80f71b0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:24.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:24.673+0000 7f5bcead3700 1 -- 172.21.15.88:0/3312970860 >> 172.21.15.88:0/3312970860 conn(0x7f5bc80f0120 msgr2=0x7f5bc80f0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:24.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:24.673+0000 7f5bcead3700 1 -- 172.21.15.88:0/3312970860 shutdown_connections 2024-02-19T22:18:24.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:24.673+0000 7f5bcead3700 1 -- 172.21.15.88:0/3312970860 wait complete. 2024-02-19T22:18:24.677 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:18:25.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:25 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:18:23.700077+0000 mgr.smithi052.ghqrtk (mgr.14162) 268 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:25.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:25 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:18:24.672908+0000 mon.smithi052 (mon.0) 272 : audit [DBG] from='client.? 172.21.15.88:0/3312970860' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:26.373 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:26.373 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:27.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:27 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:18:25.700403+0000 mgr.smithi052.ghqrtk (mgr.14162) 269 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:28.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:28.284+0000 7fc57fd39700 1 -- 172.21.15.88:0/941867462 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc5780f5410 msgr2=0x7fc5780f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:28.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:28.284+0000 7fc57fd39700 1 --2- 172.21.15.88:0/941867462 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc5780f5410 0x7fc5780f5830 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7fc56c009fb0 tx=0x7fc56c009b30).stop 2024-02-19T22:18:28.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:28.284+0000 7fc57fd39700 1 -- 172.21.15.88:0/941867462 shutdown_connections 2024-02-19T22:18:28.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:28.284+0000 7fc57fd39700 1 --2- 172.21.15.88:0/941867462 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc5780f5410 0x7fc5780f5830 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:28.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:28.284+0000 7fc57fd39700 1 -- 172.21.15.88:0/941867462 >> 172.21.15.88:0/941867462 conn(0x7fc5780f0b70 msgr2=0x7fc5780f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:28.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:28.284+0000 7fc57fd39700 1 -- 172.21.15.88:0/941867462 shutdown_connections 2024-02-19T22:18:28.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:28.284+0000 7fc57fd39700 1 -- 172.21.15.88:0/941867462 wait complete. 2024-02-19T22:18:28.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:28.285+0000 7fc57fd39700 1 Processor -- start 2024-02-19T22:18:28.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:28.285+0000 7fc57fd39700 1 -- start start 2024-02-19T22:18:28.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:28.285+0000 7fc57fd39700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc5780f5410 0x7fc578108a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:28.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:28.285+0000 7fc57fd39700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc578108f00 con 0x7fc5780f5410 2024-02-19T22:18:28.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:28.285+0000 7fc57dad5700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc5780f5410 0x7fc578108a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:28.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:28.285+0000 7fc57dad5700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc5780f5410 0x7fc578108a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:50796/0 (socket says 172.21.15.88:50796) 2024-02-19T22:18:28.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:28.285+0000 7fc57dad5700 1 -- 172.21.15.88:0/2816428254 learned_addr learned my addr 172.21.15.88:0/2816428254 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:18:28.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:28.285+0000 7fc57dad5700 1 -- 172.21.15.88:0/2816428254 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc56c005040 con 0x7fc5780f5410 2024-02-19T22:18:28.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:28.285+0000 7fc57dad5700 1 --2- 172.21.15.88:0/2816428254 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc5780f5410 0x7fc578108a10 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fc56c0089f0 tx=0x7fc56c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:28.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:28.286+0000 7fc56a7fc700 1 -- 172.21.15.88:0/2816428254 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc56c009320 con 0x7fc5780f5410 2024-02-19T22:18:28.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:28.286+0000 7fc57fd39700 1 -- 172.21.15.88:0/2816428254 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc578109100 con 0x7fc5780f5410 2024-02-19T22:18:28.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:28.286+0000 7fc56a7fc700 1 -- 172.21.15.88:0/2816428254 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc56c013380 con 0x7fc5780f5410 2024-02-19T22:18:28.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:28.286+0000 7fc56a7fc700 1 -- 172.21.15.88:0/2816428254 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc56c01ed80 con 0x7fc5780f5410 2024-02-19T22:18:28.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:28.286+0000 7fc57fd39700 1 -- 172.21.15.88:0/2816428254 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc578109590 con 0x7fc5780f5410 2024-02-19T22:18:28.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:28.286+0000 7fc56a7fc700 1 -- 172.21.15.88:0/2816428254 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc56c01e420 con 0x7fc5780f5410 2024-02-19T22:18:28.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:28.286+0000 7fc57fd39700 1 -- 172.21.15.88:0/2816428254 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc578045ad0 con 0x7fc5780f5410 2024-02-19T22:18:28.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:28.287+0000 7fc56a7fc700 1 --2- 172.21.15.88:0/2816428254 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fc56402ff50 0x7fc564032410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:28.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:28.287+0000 7fc56a7fc700 1 -- 172.21.15.88:0/2816428254 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc56c040e70 con 0x7fc5780f5410 2024-02-19T22:18:28.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:28.287+0000 7fc57d2d4700 1 --2- 172.21.15.88:0/2816428254 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fc56402ff50 0x7fc564032410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:28.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:28.287+0000 7fc57d2d4700 1 --2- 172.21.15.88:0/2816428254 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fc56402ff50 0x7fc564032410 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fc5740096a0 tx=0x7fc574006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:28.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:28.291+0000 7fc56a7fc700 1 -- 172.21.15.88:0/2816428254 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc56c00e2a0 con 0x7fc5780f5410 2024-02-19T22:18:28.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:28.494+0000 7fc57fd39700 1 -- 172.21.15.88:0/2816428254 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc57810a440 con 0x7fc5780f5410 2024-02-19T22:18:28.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:28.495+0000 7fc56a7fc700 1 -- 172.21.15.88:0/2816428254 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc56c00e2a0 con 0x7fc5780f5410 2024-02-19T22:18:28.498 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:18:28.498 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:28.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:28.497+0000 7fc57fd39700 1 -- 172.21.15.88:0/2816428254 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fc56402ff50 msgr2=0x7fc564032410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:28.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:28.497+0000 7fc57fd39700 1 --2- 172.21.15.88:0/2816428254 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fc56402ff50 0x7fc564032410 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fc5740096a0 tx=0x7fc574006b40).stop 2024-02-19T22:18:28.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:28.497+0000 7fc57fd39700 1 -- 172.21.15.88:0/2816428254 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc5780f5410 msgr2=0x7fc578108a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:28.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:28.497+0000 7fc57fd39700 1 --2- 172.21.15.88:0/2816428254 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc5780f5410 0x7fc578108a10 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fc56c0089f0 tx=0x7fc56c00b040).stop 2024-02-19T22:18:28.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:28.497+0000 7fc57fd39700 1 -- 172.21.15.88:0/2816428254 shutdown_connections 2024-02-19T22:18:28.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:28.497+0000 7fc57fd39700 1 --2- 172.21.15.88:0/2816428254 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fc56402ff50 0x7fc564032410 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:28.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:28.497+0000 7fc57fd39700 1 --2- 172.21.15.88:0/2816428254 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc5780f5410 0x7fc578108a10 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:28.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:28.497+0000 7fc57fd39700 1 -- 172.21.15.88:0/2816428254 >> 172.21.15.88:0/2816428254 conn(0x7fc5780f0b70 msgr2=0x7fc5780f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:28.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:28.497+0000 7fc57fd39700 1 -- 172.21.15.88:0/2816428254 shutdown_connections 2024-02-19T22:18:28.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:28.497+0000 7fc57fd39700 1 -- 172.21.15.88:0/2816428254 wait complete. 2024-02-19T22:18:28.500 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:18:29.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:29 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:18:27.700765+0000 mgr.smithi052.ghqrtk (mgr.14162) 270 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:29.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:29 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:18:28.496595+0000 mon.smithi052 (mon.0) 273 : audit [DBG] from='client.? 172.21.15.88:0/2816428254' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:30.236 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:30.236 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:31.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:31 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:18:29.701082+0000 mgr.smithi052.ghqrtk (mgr.14162) 271 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:32.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:32.084+0000 7f7911e3a700 1 -- 172.21.15.88:0/1523670492 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f790c060280 msgr2=0x7f790c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:32.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:32.084+0000 7f7911e3a700 1 --2- 172.21.15.88:0/1523670492 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f790c060280 0x7f790c0f90f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f7900009fb0 tx=0x7f7900009b30).stop 2024-02-19T22:18:32.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:32.084+0000 7f7911e3a700 1 -- 172.21.15.88:0/1523670492 shutdown_connections 2024-02-19T22:18:32.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:32.084+0000 7f7911e3a700 1 --2- 172.21.15.88:0/1523670492 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f790c060280 0x7f790c0f90f0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:32.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:32.084+0000 7f7911e3a700 1 -- 172.21.15.88:0/1523670492 >> 172.21.15.88:0/1523670492 conn(0x7f790c0f0b70 msgr2=0x7f790c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:32.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:32.084+0000 7f7911e3a700 1 -- 172.21.15.88:0/1523670492 shutdown_connections 2024-02-19T22:18:32.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:32.084+0000 7f7911e3a700 1 -- 172.21.15.88:0/1523670492 wait complete. 2024-02-19T22:18:32.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:32.085+0000 7f7911e3a700 1 Processor -- start 2024-02-19T22:18:32.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:32.085+0000 7f7911e3a700 1 -- start start 2024-02-19T22:18:32.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:32.085+0000 7f7911e3a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f790c060280 0x7f790c106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:32.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:32.085+0000 7f7911e3a700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f790c106c70 con 0x7f790c060280 2024-02-19T22:18:32.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:32.086+0000 7f790b7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f790c060280 0x7f790c106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:32.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:32.086+0000 7f790b7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f790c060280 0x7f790c106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:50812/0 (socket says 172.21.15.88:50812) 2024-02-19T22:18:32.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:32.086+0000 7f790b7fe700 1 -- 172.21.15.88:0/1010759512 learned_addr learned my addr 172.21.15.88:0/1010759512 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:18:32.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:32.086+0000 7f790b7fe700 1 -- 172.21.15.88:0/1010759512 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7900005040 con 0x7f790c060280 2024-02-19T22:18:32.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:32.086+0000 7f790b7fe700 1 --2- 172.21.15.88:0/1010759512 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f790c060280 0x7f790c106780 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f79000089f0 tx=0x7f790000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:32.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:32.086+0000 7f78f3fff700 1 -- 172.21.15.88:0/1010759512 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7900009320 con 0x7f790c060280 2024-02-19T22:18:32.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:32.086+0000 7f7911e3a700 1 -- 172.21.15.88:0/1010759512 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f790c106e70 con 0x7f790c060280 2024-02-19T22:18:32.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:32.086+0000 7f78f3fff700 1 -- 172.21.15.88:0/1010759512 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7900013380 con 0x7f790c060280 2024-02-19T22:18:32.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:32.086+0000 7f78f3fff700 1 -- 172.21.15.88:0/1010759512 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7900014e20 con 0x7f790c060280 2024-02-19T22:18:32.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:32.086+0000 7f7911e3a700 1 -- 172.21.15.88:0/1010759512 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f790c107300 con 0x7f790c060280 2024-02-19T22:18:32.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:32.087+0000 7f78f3fff700 1 -- 172.21.15.88:0/1010759512 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f79000095f0 con 0x7f790c060280 2024-02-19T22:18:32.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:32.087+0000 7f7911e3a700 1 -- 172.21.15.88:0/1010759512 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f790c045ad0 con 0x7f790c060280 2024-02-19T22:18:32.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:32.087+0000 7f78f3fff700 1 --2- 172.21.15.88:0/1010759512 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f78f402ff10 0x7f78f40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:32.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:32.087+0000 7f78f3fff700 1 -- 172.21.15.88:0/1010759512 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7900040ef0 con 0x7f790c060280 2024-02-19T22:18:32.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:32.088+0000 7f790affd700 1 --2- 172.21.15.88:0/1010759512 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f78f402ff10 0x7f78f40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:32.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:32.088+0000 7f790affd700 1 --2- 172.21.15.88:0/1010759512 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f78f402ff10 0x7f78f40323d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f78fc0096a0 tx=0x7f78fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:32.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:32.093+0000 7f78f3fff700 1 -- 172.21.15.88:0/1010759512 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7900014420 con 0x7f790c060280 2024-02-19T22:18:32.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:32.297+0000 7f7911e3a700 1 -- 172.21.15.88:0/1010759512 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f790c108150 con 0x7f790c060280 2024-02-19T22:18:32.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:32.298+0000 7f78f3fff700 1 -- 172.21.15.88:0/1010759512 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7900010230 con 0x7f790c060280 2024-02-19T22:18:32.300 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:18:32.300 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:32.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:32.299+0000 7f7911e3a700 1 -- 172.21.15.88:0/1010759512 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f78f402ff10 msgr2=0x7f78f40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:32.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:32.299+0000 7f7911e3a700 1 --2- 172.21.15.88:0/1010759512 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f78f402ff10 0x7f78f40323d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f78fc0096a0 tx=0x7f78fc006b40).stop 2024-02-19T22:18:32.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:32.299+0000 7f7911e3a700 1 -- 172.21.15.88:0/1010759512 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f790c060280 msgr2=0x7f790c106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:32.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:32.299+0000 7f7911e3a700 1 --2- 172.21.15.88:0/1010759512 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f790c060280 0x7f790c106780 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f79000089f0 tx=0x7f790000b040).stop 2024-02-19T22:18:32.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:32.299+0000 7f7911e3a700 1 -- 172.21.15.88:0/1010759512 shutdown_connections 2024-02-19T22:18:32.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:32.299+0000 7f7911e3a700 1 --2- 172.21.15.88:0/1010759512 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f78f402ff10 0x7f78f40323d0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:32.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:32.299+0000 7f7911e3a700 1 --2- 172.21.15.88:0/1010759512 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f790c060280 0x7f790c106780 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:32.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:32.299+0000 7f7911e3a700 1 -- 172.21.15.88:0/1010759512 >> 172.21.15.88:0/1010759512 conn(0x7f790c0f0b70 msgr2=0x7f790c0f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:32.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:32.299+0000 7f7911e3a700 1 -- 172.21.15.88:0/1010759512 shutdown_connections 2024-02-19T22:18:32.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:32.299+0000 7f7911e3a700 1 -- 172.21.15.88:0/1010759512 wait complete. 2024-02-19T22:18:32.303 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:18:33.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:33 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:18:31.701411+0000 mgr.smithi052.ghqrtk (mgr.14162) 272 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:33.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:33 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:18:32.298882+0000 mon.smithi052 (mon.0) 274 : audit [DBG] from='client.? 172.21.15.88:0/1010759512' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:34.044 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:34.044 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:35.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:35 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:18:33.701777+0000 mgr.smithi052.ghqrtk (mgr.14162) 273 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:36.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:36.024+0000 7f3b29a9a700 1 -- 172.21.15.88:0/1556396538 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3b240f6510 msgr2=0x7f3b240f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:36.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:36.024+0000 7f3b29a9a700 1 --2- 172.21.15.88:0/1556396538 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3b240f6510 0x7f3b240f6930 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f3b18009fb0 tx=0x7f3b18009b30).stop 2024-02-19T22:18:36.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:36.024+0000 7f3b29a9a700 1 -- 172.21.15.88:0/1556396538 shutdown_connections 2024-02-19T22:18:36.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:36.024+0000 7f3b29a9a700 1 --2- 172.21.15.88:0/1556396538 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3b240f6510 0x7f3b240f6930 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:36.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:36.024+0000 7f3b29a9a700 1 -- 172.21.15.88:0/1556396538 >> 172.21.15.88:0/1556396538 conn(0x7f3b240f1c70 msgr2=0x7f3b240f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:36.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:36.024+0000 7f3b29a9a700 1 -- 172.21.15.88:0/1556396538 shutdown_connections 2024-02-19T22:18:36.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:36.024+0000 7f3b29a9a700 1 -- 172.21.15.88:0/1556396538 wait complete. 2024-02-19T22:18:36.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:36.025+0000 7f3b29a9a700 1 Processor -- start 2024-02-19T22:18:36.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:36.025+0000 7f3b29a9a700 1 -- start start 2024-02-19T22:18:36.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:36.025+0000 7f3b29a9a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3b240f6510 0x7f3b241067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:36.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:36.025+0000 7f3b29a9a700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b24106cd0 con 0x7f3b240f6510 2024-02-19T22:18:36.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:36.025+0000 7f3b237fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3b240f6510 0x7f3b241067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:36.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:36.025+0000 7f3b237fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3b240f6510 0x7f3b241067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:40184/0 (socket says 172.21.15.88:40184) 2024-02-19T22:18:36.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:36.025+0000 7f3b237fe700 1 -- 172.21.15.88:0/578718978 learned_addr learned my addr 172.21.15.88:0/578718978 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:18:36.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:36.026+0000 7f3b237fe700 1 -- 172.21.15.88:0/578718978 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b18005040 con 0x7f3b240f6510 2024-02-19T22:18:36.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:36.026+0000 7f3b237fe700 1 --2- 172.21.15.88:0/578718978 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3b240f6510 0x7f3b241067e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f3b18005010 tx=0x7f3b1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:36.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:36.026+0000 7f3b0bfff700 1 -- 172.21.15.88:0/578718978 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b18009320 con 0x7f3b240f6510 2024-02-19T22:18:36.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:36.026+0000 7f3b29a9a700 1 -- 172.21.15.88:0/578718978 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b24106ed0 con 0x7f3b240f6510 2024-02-19T22:18:36.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:36.026+0000 7f3b0bfff700 1 -- 172.21.15.88:0/578718978 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3b18013380 con 0x7f3b240f6510 2024-02-19T22:18:36.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:36.026+0000 7f3b0bfff700 1 -- 172.21.15.88:0/578718978 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b18014e50 con 0x7f3b240f6510 2024-02-19T22:18:36.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:36.026+0000 7f3b29a9a700 1 -- 172.21.15.88:0/578718978 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b24107360 con 0x7f3b240f6510 2024-02-19T22:18:36.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:36.027+0000 7f3b0bfff700 1 -- 172.21.15.88:0/578718978 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3b180263a0 con 0x7f3b240f6510 2024-02-19T22:18:36.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:36.027+0000 7f3b0bfff700 1 --2- 172.21.15.88:0/578718978 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f3b0c02ff10 0x7f3b0c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:36.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:36.027+0000 7f3b29a9a700 1 -- 172.21.15.88:0/578718978 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b24045ad0 con 0x7f3b240f6510 2024-02-19T22:18:36.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:36.027+0000 7f3b0bfff700 1 -- 172.21.15.88:0/578718978 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3b18040ff0 con 0x7f3b240f6510 2024-02-19T22:18:36.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:36.027+0000 7f3b22ffd700 1 --2- 172.21.15.88:0/578718978 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f3b0c02ff10 0x7f3b0c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:36.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:36.028+0000 7f3b22ffd700 1 --2- 172.21.15.88:0/578718978 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f3b0c02ff10 0x7f3b0c0323d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f3b140096a0 tx=0x7f3b14006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:36.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:36.032+0000 7f3b0bfff700 1 -- 172.21.15.88:0/578718978 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3b18012920 con 0x7f3b240f6510 2024-02-19T22:18:36.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:36.236+0000 7f3b29a9a700 1 -- 172.21.15.88:0/578718978 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3b24108210 con 0x7f3b240f6510 2024-02-19T22:18:36.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:36.236+0000 7f3b0bfff700 1 -- 172.21.15.88:0/578718978 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3b1802b020 con 0x7f3b240f6510 2024-02-19T22:18:36.238 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:18:36.238 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:36.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:36.237+0000 7f3b29a9a700 1 -- 172.21.15.88:0/578718978 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f3b0c02ff10 msgr2=0x7f3b0c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:36.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:36.237+0000 7f3b29a9a700 1 --2- 172.21.15.88:0/578718978 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f3b0c02ff10 0x7f3b0c0323d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f3b140096a0 tx=0x7f3b14006b40).stop 2024-02-19T22:18:36.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:36.237+0000 7f3b29a9a700 1 -- 172.21.15.88:0/578718978 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3b240f6510 msgr2=0x7f3b241067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:36.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:36.237+0000 7f3b29a9a700 1 --2- 172.21.15.88:0/578718978 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3b240f6510 0x7f3b241067e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f3b18005010 tx=0x7f3b1800b040).stop 2024-02-19T22:18:36.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:36.237+0000 7f3b29a9a700 1 -- 172.21.15.88:0/578718978 shutdown_connections 2024-02-19T22:18:36.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:36.237+0000 7f3b29a9a700 1 --2- 172.21.15.88:0/578718978 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f3b0c02ff10 0x7f3b0c0323d0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:36.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:36.237+0000 7f3b29a9a700 1 --2- 172.21.15.88:0/578718978 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3b240f6510 0x7f3b241067e0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:36.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:36.237+0000 7f3b29a9a700 1 -- 172.21.15.88:0/578718978 >> 172.21.15.88:0/578718978 conn(0x7f3b240f1c70 msgr2=0x7f3b240f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:36.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:36.238+0000 7f3b29a9a700 1 -- 172.21.15.88:0/578718978 shutdown_connections 2024-02-19T22:18:36.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:36.238+0000 7f3b29a9a700 1 -- 172.21.15.88:0/578718978 wait complete. 2024-02-19T22:18:36.240 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:18:36.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:36 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:18:36.236531+0000 mon.smithi052 (mon.0) 275 : audit [DBG] from='client.? 172.21.15.88:0/578718978' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:37.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:37 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:18:35.702164+0000 mgr.smithi052.ghqrtk (mgr.14162) 274 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:37.872 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:37.873 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:39.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:18:37.702540+0000 mgr.smithi052.ghqrtk (mgr.14162) 275 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:39.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:39.683+0000 7fa9fe533700 1 -- 172.21.15.88:0/1523822391 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9f80f6510 msgr2=0x7fa9f80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:39.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:39.683+0000 7fa9fe533700 1 --2- 172.21.15.88:0/1523822391 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9f80f6510 0x7fa9f80f6930 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7fa9e8009fb0 tx=0x7fa9e8009b30).stop 2024-02-19T22:18:39.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:39.683+0000 7fa9fe533700 1 -- 172.21.15.88:0/1523822391 shutdown_connections 2024-02-19T22:18:39.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:39.683+0000 7fa9fe533700 1 --2- 172.21.15.88:0/1523822391 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9f80f6510 0x7fa9f80f6930 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:39.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:39.683+0000 7fa9fe533700 1 -- 172.21.15.88:0/1523822391 >> 172.21.15.88:0/1523822391 conn(0x7fa9f80f1c70 msgr2=0x7fa9f80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:39.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:39.683+0000 7fa9fe533700 1 -- 172.21.15.88:0/1523822391 shutdown_connections 2024-02-19T22:18:39.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:39.683+0000 7fa9fe533700 1 -- 172.21.15.88:0/1523822391 wait complete. 2024-02-19T22:18:39.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:39.684+0000 7fa9fe533700 1 Processor -- start 2024-02-19T22:18:39.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:39.684+0000 7fa9fe533700 1 -- start start 2024-02-19T22:18:39.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:39.684+0000 7fa9fe533700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9f80f6510 0x7fa9f81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:39.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:39.684+0000 7fa9fe533700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f8106cd0 con 0x7fa9f80f6510 2024-02-19T22:18:39.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:39.684+0000 7fa9f7fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9f80f6510 0x7fa9f81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:39.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:39.684+0000 7fa9f7fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9f80f6510 0x7fa9f81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:40200/0 (socket says 172.21.15.88:40200) 2024-02-19T22:18:39.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:39.684+0000 7fa9f7fff700 1 -- 172.21.15.88:0/351130081 learned_addr learned my addr 172.21.15.88:0/351130081 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:18:39.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:39.685+0000 7fa9f7fff700 1 -- 172.21.15.88:0/351130081 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e8005040 con 0x7fa9f80f6510 2024-02-19T22:18:39.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:39.685+0000 7fa9f7fff700 1 --2- 172.21.15.88:0/351130081 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9f80f6510 0x7fa9f81067e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7fa9e80089f0 tx=0x7fa9e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:39.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:39.685+0000 7fa9f4ff9700 1 -- 172.21.15.88:0/351130081 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9e8009320 con 0x7fa9f80f6510 2024-02-19T22:18:39.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:39.685+0000 7fa9f4ff9700 1 -- 172.21.15.88:0/351130081 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa9e8013380 con 0x7fa9f80f6510 2024-02-19T22:18:39.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:39.685+0000 7fa9fe533700 1 -- 172.21.15.88:0/351130081 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9f8106ed0 con 0x7fa9f80f6510 2024-02-19T22:18:39.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:39.685+0000 7fa9f4ff9700 1 -- 172.21.15.88:0/351130081 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9e801ed80 con 0x7fa9f80f6510 2024-02-19T22:18:39.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:39.685+0000 7fa9fe533700 1 -- 172.21.15.88:0/351130081 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9f8107360 con 0x7fa9f80f6510 2024-02-19T22:18:39.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:39.686+0000 7fa9f4ff9700 1 -- 172.21.15.88:0/351130081 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa9e8014420 con 0x7fa9f80f6510 2024-02-19T22:18:39.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:39.686+0000 7fa9f4ff9700 1 --2- 172.21.15.88:0/351130081 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa9e002ff60 0x7fa9e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:39.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:39.686+0000 7fa9f4ff9700 1 -- 172.21.15.88:0/351130081 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa9e8040e00 con 0x7fa9f80f6510 2024-02-19T22:18:39.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:39.686+0000 7fa9fe533700 1 -- 172.21.15.88:0/351130081 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9f8045ad0 con 0x7fa9f80f6510 2024-02-19T22:18:39.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:39.686+0000 7fa9f77fe700 1 --2- 172.21.15.88:0/351130081 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa9e002ff60 0x7fa9e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:39.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:39.686+0000 7fa9f77fe700 1 --2- 172.21.15.88:0/351130081 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa9e002ff60 0x7fa9e0032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fa9ec0096a0 tx=0x7fa9ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:39.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:39.691+0000 7fa9f4ff9700 1 -- 172.21.15.88:0/351130081 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa9e801c910 con 0x7fa9f80f6510 2024-02-19T22:18:39.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:39.896+0000 7fa9fe533700 1 -- 172.21.15.88:0/351130081 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa9f80fc190 con 0x7fa9f80f6510 2024-02-19T22:18:39.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:39.897+0000 7fa9f4ff9700 1 -- 172.21.15.88:0/351130081 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa9e801c320 con 0x7fa9f80f6510 2024-02-19T22:18:39.899 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:18:39.899 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:39.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:39.899+0000 7fa9fe533700 1 -- 172.21.15.88:0/351130081 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa9e002ff60 msgr2=0x7fa9e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:39.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:39.899+0000 7fa9fe533700 1 --2- 172.21.15.88:0/351130081 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa9e002ff60 0x7fa9e0032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fa9ec0096a0 tx=0x7fa9ec006b40).stop 2024-02-19T22:18:39.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:39.899+0000 7fa9fe533700 1 -- 172.21.15.88:0/351130081 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9f80f6510 msgr2=0x7fa9f81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:39.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:39.899+0000 7fa9fe533700 1 --2- 172.21.15.88:0/351130081 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9f80f6510 0x7fa9f81067e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7fa9e80089f0 tx=0x7fa9e800b040).stop 2024-02-19T22:18:39.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:39.899+0000 7fa9fe533700 1 -- 172.21.15.88:0/351130081 shutdown_connections 2024-02-19T22:18:39.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:39.899+0000 7fa9fe533700 1 --2- 172.21.15.88:0/351130081 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa9e002ff60 0x7fa9e0032420 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:39.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:39.899+0000 7fa9fe533700 1 --2- 172.21.15.88:0/351130081 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9f80f6510 0x7fa9f81067e0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:39.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:39.899+0000 7fa9fe533700 1 -- 172.21.15.88:0/351130081 >> 172.21.15.88:0/351130081 conn(0x7fa9f80f1c70 msgr2=0x7fa9f80f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:39.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:39.899+0000 7fa9fe533700 1 -- 172.21.15.88:0/351130081 shutdown_connections 2024-02-19T22:18:39.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:39.899+0000 7fa9fe533700 1 -- 172.21.15.88:0/351130081 wait complete. 2024-02-19T22:18:39.901 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:18:40.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:40 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:18:39.897551+0000 mon.smithi052 (mon.0) 276 : audit [DBG] from='client.? 172.21.15.88:0/351130081' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:41.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:41 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:18:39.702867+0000 mgr.smithi052.ghqrtk (mgr.14162) 276 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:41.669 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:41.670 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:43.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:43.487+0000 7f6b36a20700 1 -- 172.21.15.88:0/4126799268 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6b300f5410 msgr2=0x7f6b300f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:43.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:43.487+0000 7f6b36a20700 1 --2- 172.21.15.88:0/4126799268 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6b300f5410 0x7f6b300f5830 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f6b20009fb0 tx=0x7f6b20009b30).stop 2024-02-19T22:18:43.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:43.488+0000 7f6b36a20700 1 -- 172.21.15.88:0/4126799268 shutdown_connections 2024-02-19T22:18:43.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:43.488+0000 7f6b36a20700 1 --2- 172.21.15.88:0/4126799268 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6b300f5410 0x7f6b300f5830 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:43.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:43.488+0000 7f6b36a20700 1 -- 172.21.15.88:0/4126799268 >> 172.21.15.88:0/4126799268 conn(0x7f6b300f0b70 msgr2=0x7f6b300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:43.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:43.488+0000 7f6b36a20700 1 -- 172.21.15.88:0/4126799268 shutdown_connections 2024-02-19T22:18:43.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:43.488+0000 7f6b36a20700 1 -- 172.21.15.88:0/4126799268 wait complete. 2024-02-19T22:18:43.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:43.488+0000 7f6b36a20700 1 Processor -- start 2024-02-19T22:18:43.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:43.488+0000 7f6b36a20700 1 -- start start 2024-02-19T22:18:43.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:43.488+0000 7f6b36a20700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6b300f5410 0x7f6b30109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:43.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:43.488+0000 7f6b36a20700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b30060220 con 0x7f6b300f5410 2024-02-19T22:18:43.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:43.489+0000 7f6b2ffff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6b300f5410 0x7f6b30109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:43.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:43.489+0000 7f6b2ffff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6b300f5410 0x7f6b30109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:53690/0 (socket says 172.21.15.88:53690) 2024-02-19T22:18:43.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:43.489+0000 7f6b2ffff700 1 -- 172.21.15.88:0/996654490 learned_addr learned my addr 172.21.15.88:0/996654490 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:18:43.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:43.489+0000 7f6b2ffff700 1 -- 172.21.15.88:0/996654490 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b20005040 con 0x7f6b300f5410 2024-02-19T22:18:43.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:43.489+0000 7f6b2ffff700 1 --2- 172.21.15.88:0/996654490 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6b300f5410 0x7f6b30109500 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f6b20005010 tx=0x7f6b2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:43.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:43.489+0000 7f6b2cff9700 1 -- 172.21.15.88:0/996654490 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b20009320 con 0x7f6b300f5410 2024-02-19T22:18:43.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:43.489+0000 7f6b36a20700 1 -- 172.21.15.88:0/996654490 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b30060420 con 0x7f6b300f5410 2024-02-19T22:18:43.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:43.489+0000 7f6b2cff9700 1 -- 172.21.15.88:0/996654490 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6b20013380 con 0x7f6b300f5410 2024-02-19T22:18:43.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:43.489+0000 7f6b2cff9700 1 -- 172.21.15.88:0/996654490 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b20014e50 con 0x7f6b300f5410 2024-02-19T22:18:43.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:43.489+0000 7f6b36a20700 1 -- 172.21.15.88:0/996654490 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b30109bb0 con 0x7f6b300f5410 2024-02-19T22:18:43.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:43.490+0000 7f6b2cff9700 1 -- 172.21.15.88:0/996654490 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6b20014420 con 0x7f6b300f5410 2024-02-19T22:18:43.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:43.490+0000 7f6b36a20700 1 -- 172.21.15.88:0/996654490 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b30059100 con 0x7f6b300f5410 2024-02-19T22:18:43.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:43.490+0000 7f6b2cff9700 1 --2- 172.21.15.88:0/996654490 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f6b1802ff10 0x7f6b180323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:43.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:43.490+0000 7f6b2cff9700 1 -- 172.21.15.88:0/996654490 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6b20040e30 con 0x7f6b300f5410 2024-02-19T22:18:43.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:43.490+0000 7f6b2f7fe700 1 --2- 172.21.15.88:0/996654490 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f6b1802ff10 0x7f6b180323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:43.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:43.491+0000 7f6b2f7fe700 1 --2- 172.21.15.88:0/996654490 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f6b1802ff10 0x7f6b180323d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f6b240096a0 tx=0x7f6b24006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:43.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:43.495+0000 7f6b2cff9700 1 -- 172.21.15.88:0/996654490 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6b200095f0 con 0x7f6b300f5410 2024-02-19T22:18:43.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:43 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:18:41.703138+0000 mgr.smithi052.ghqrtk (mgr.14162) 277 : cluster [DBG] pgmap v263: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:43.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:43.698+0000 7f6b36a20700 1 -- 172.21.15.88:0/996654490 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6b30045ad0 con 0x7f6b300f5410 2024-02-19T22:18:43.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:43.699+0000 7f6b2cff9700 1 -- 172.21.15.88:0/996654490 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6b2001db70 con 0x7f6b300f5410 2024-02-19T22:18:43.700 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:18:43.701 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:43.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:43.700+0000 7f6b36a20700 1 -- 172.21.15.88:0/996654490 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f6b1802ff10 msgr2=0x7f6b180323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:43.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:43.700+0000 7f6b36a20700 1 --2- 172.21.15.88:0/996654490 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f6b1802ff10 0x7f6b180323d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f6b240096a0 tx=0x7f6b24006b40).stop 2024-02-19T22:18:43.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:43.700+0000 7f6b36a20700 1 -- 172.21.15.88:0/996654490 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6b300f5410 msgr2=0x7f6b30109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:43.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:43.700+0000 7f6b36a20700 1 --2- 172.21.15.88:0/996654490 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6b300f5410 0x7f6b30109500 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f6b20005010 tx=0x7f6b2000b040).stop 2024-02-19T22:18:43.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:43.700+0000 7f6b36a20700 1 -- 172.21.15.88:0/996654490 shutdown_connections 2024-02-19T22:18:43.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:43.700+0000 7f6b36a20700 1 --2- 172.21.15.88:0/996654490 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f6b1802ff10 0x7f6b180323d0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:43.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:43.701+0000 7f6b36a20700 1 --2- 172.21.15.88:0/996654490 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6b300f5410 0x7f6b30109500 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:43.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:43.701+0000 7f6b36a20700 1 -- 172.21.15.88:0/996654490 >> 172.21.15.88:0/996654490 conn(0x7f6b300f0b70 msgr2=0x7f6b300f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:43.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:43.701+0000 7f6b36a20700 1 -- 172.21.15.88:0/996654490 shutdown_connections 2024-02-19T22:18:43.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:43.701+0000 7f6b36a20700 1 -- 172.21.15.88:0/996654490 wait complete. 2024-02-19T22:18:43.703 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:18:44.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:44 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:18:43.699431+0000 mon.smithi052 (mon.0) 277 : audit [DBG] from='client.? 172.21.15.88:0/996654490' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:44.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:44 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:18:43.754025+0000 mon.smithi052 (mon.0) 278 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi052.ghqrtk/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:18:44.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:44 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:18:43.756663+0000 mon.smithi052 (mon.0) 279 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi052.ghqrtk/trash_purge_schedule"}]: dispatch 2024-02-19T22:18:45.426 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:45.426 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:45.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:45 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:18:43.703412+0000 mgr.smithi052.ghqrtk (mgr.14162) 278 : cluster [DBG] pgmap v264: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:47.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:47.108+0000 7fa3ff085700 1 -- 172.21.15.88:0/777325114 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa3f80f5410 msgr2=0x7fa3f80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:47.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:47.108+0000 7fa3ff085700 1 --2- 172.21.15.88:0/777325114 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa3f80f5410 0x7fa3f80f5830 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7fa3ec009fb0 tx=0x7fa3ec009b30).stop 2024-02-19T22:18:47.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:47.108+0000 7fa3ff085700 1 -- 172.21.15.88:0/777325114 shutdown_connections 2024-02-19T22:18:47.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:47.108+0000 7fa3ff085700 1 --2- 172.21.15.88:0/777325114 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa3f80f5410 0x7fa3f80f5830 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:47.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:47.108+0000 7fa3ff085700 1 -- 172.21.15.88:0/777325114 >> 172.21.15.88:0/777325114 conn(0x7fa3f80f0b70 msgr2=0x7fa3f80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:47.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:47.108+0000 7fa3ff085700 1 -- 172.21.15.88:0/777325114 shutdown_connections 2024-02-19T22:18:47.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:47.108+0000 7fa3ff085700 1 -- 172.21.15.88:0/777325114 wait complete. 2024-02-19T22:18:47.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:47.108+0000 7fa3ff085700 1 Processor -- start 2024-02-19T22:18:47.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:47.108+0000 7fa3ff085700 1 -- start start 2024-02-19T22:18:47.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:47.109+0000 7fa3ff085700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa3f80f5410 0x7fa3f81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:47.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:47.109+0000 7fa3ff085700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f8106cd0 con 0x7fa3f80f5410 2024-02-19T22:18:47.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:47.109+0000 7fa3fce21700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa3f80f5410 0x7fa3f81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:47.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:47.109+0000 7fa3fce21700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa3f80f5410 0x7fa3f81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:53708/0 (socket says 172.21.15.88:53708) 2024-02-19T22:18:47.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:47.109+0000 7fa3fce21700 1 -- 172.21.15.88:0/3036140382 learned_addr learned my addr 172.21.15.88:0/3036140382 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:18:47.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:47.109+0000 7fa3fce21700 1 -- 172.21.15.88:0/3036140382 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3ec005040 con 0x7fa3f80f5410 2024-02-19T22:18:47.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:47.109+0000 7fa3fce21700 1 --2- 172.21.15.88:0/3036140382 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa3f80f5410 0x7fa3f81067e0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7fa3ec005010 tx=0x7fa3ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:47.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:47.109+0000 7fa3f57fa700 1 -- 172.21.15.88:0/3036140382 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3ec009320 con 0x7fa3f80f5410 2024-02-19T22:18:47.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:47.110+0000 7fa3ff085700 1 -- 172.21.15.88:0/3036140382 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3f8106ed0 con 0x7fa3f80f5410 2024-02-19T22:18:47.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:47.110+0000 7fa3f57fa700 1 -- 172.21.15.88:0/3036140382 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa3ec013380 con 0x7fa3f80f5410 2024-02-19T22:18:47.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:47.110+0000 7fa3f57fa700 1 -- 172.21.15.88:0/3036140382 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3ec014e50 con 0x7fa3f80f5410 2024-02-19T22:18:47.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:47.110+0000 7fa3ff085700 1 -- 172.21.15.88:0/3036140382 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3f8107360 con 0x7fa3f80f5410 2024-02-19T22:18:47.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:47.110+0000 7fa3f57fa700 1 -- 172.21.15.88:0/3036140382 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa3ec014420 con 0x7fa3f80f5410 2024-02-19T22:18:47.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:47.110+0000 7fa3ff085700 1 -- 172.21.15.88:0/3036140382 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3f8045ad0 con 0x7fa3f80f5410 2024-02-19T22:18:47.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:47.111+0000 7fa3f57fa700 1 --2- 172.21.15.88:0/3036140382 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa3e802ff10 0x7fa3e80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:47.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:47.111+0000 7fa3f57fa700 1 -- 172.21.15.88:0/3036140382 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa3ec040f40 con 0x7fa3f80f5410 2024-02-19T22:18:47.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:47.111+0000 7fa3f7fff700 1 --2- 172.21.15.88:0/3036140382 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa3e802ff10 0x7fa3e80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:47.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:47.111+0000 7fa3f7fff700 1 --2- 172.21.15.88:0/3036140382 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa3e802ff10 0x7fa3e80323d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fa3e40096a0 tx=0x7fa3e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:47.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:47.115+0000 7fa3f57fa700 1 -- 172.21.15.88:0/3036140382 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa3ec0095f0 con 0x7fa3f80f5410 2024-02-19T22:18:47.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:47.319+0000 7fa3ff085700 1 -- 172.21.15.88:0/3036140382 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa3f8108210 con 0x7fa3f80f5410 2024-02-19T22:18:47.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:47.320+0000 7fa3f57fa700 1 -- 172.21.15.88:0/3036140382 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa3ec01db70 con 0x7fa3f80f5410 2024-02-19T22:18:47.321 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:18:47.321 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:47.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:47.321+0000 7fa3ff085700 1 -- 172.21.15.88:0/3036140382 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa3e802ff10 msgr2=0x7fa3e80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:47.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:47.321+0000 7fa3ff085700 1 --2- 172.21.15.88:0/3036140382 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa3e802ff10 0x7fa3e80323d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fa3e40096a0 tx=0x7fa3e4006b40).stop 2024-02-19T22:18:47.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:47.321+0000 7fa3ff085700 1 -- 172.21.15.88:0/3036140382 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa3f80f5410 msgr2=0x7fa3f81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:47.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:47.321+0000 7fa3ff085700 1 --2- 172.21.15.88:0/3036140382 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa3f80f5410 0x7fa3f81067e0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7fa3ec005010 tx=0x7fa3ec00b040).stop 2024-02-19T22:18:47.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:47.321+0000 7fa3ff085700 1 -- 172.21.15.88:0/3036140382 shutdown_connections 2024-02-19T22:18:47.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:47.321+0000 7fa3ff085700 1 --2- 172.21.15.88:0/3036140382 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa3e802ff10 0x7fa3e80323d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:47.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:47.321+0000 7fa3ff085700 1 --2- 172.21.15.88:0/3036140382 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa3f80f5410 0x7fa3f81067e0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:47.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:47.321+0000 7fa3ff085700 1 -- 172.21.15.88:0/3036140382 >> 172.21.15.88:0/3036140382 conn(0x7fa3f80f0b70 msgr2=0x7fa3f80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:47.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:47.321+0000 7fa3ff085700 1 -- 172.21.15.88:0/3036140382 shutdown_connections 2024-02-19T22:18:47.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:47.321+0000 7fa3ff085700 1 -- 172.21.15.88:0/3036140382 wait complete. 2024-02-19T22:18:47.324 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:18:47.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:47 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:18:45.703694+0000 mgr.smithi052.ghqrtk (mgr.14162) 279 : cluster [DBG] pgmap v265: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:48.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:48 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:18:47.320146+0000 mon.smithi052 (mon.0) 280 : audit [DBG] from='client.? 172.21.15.88:0/3036140382' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:49.099 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:49.100 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:49.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:49 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:18:47.704066+0000 mgr.smithi052.ghqrtk (mgr.14162) 280 : cluster [DBG] pgmap v266: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:50.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:50.919+0000 7f5aa718b700 1 -- 172.21.15.88:0/2469693486 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5aa00f6510 msgr2=0x7f5aa00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:50.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:50.919+0000 7f5aa718b700 1 --2- 172.21.15.88:0/2469693486 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5aa00f6510 0x7f5aa00f6930 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f5a94009fb0 tx=0x7f5a94009b30).stop 2024-02-19T22:18:50.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:50.919+0000 7f5aa718b700 1 -- 172.21.15.88:0/2469693486 shutdown_connections 2024-02-19T22:18:50.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:50.919+0000 7f5aa718b700 1 --2- 172.21.15.88:0/2469693486 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5aa00f6510 0x7f5aa00f6930 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:50.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:50.919+0000 7f5aa718b700 1 -- 172.21.15.88:0/2469693486 >> 172.21.15.88:0/2469693486 conn(0x7f5aa00f1c70 msgr2=0x7f5aa00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:50.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:50.919+0000 7f5aa718b700 1 -- 172.21.15.88:0/2469693486 shutdown_connections 2024-02-19T22:18:50.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:50.919+0000 7f5aa718b700 1 -- 172.21.15.88:0/2469693486 wait complete. 2024-02-19T22:18:50.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:50.919+0000 7f5aa718b700 1 Processor -- start 2024-02-19T22:18:50.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:50.919+0000 7f5aa718b700 1 -- start start 2024-02-19T22:18:50.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:50.920+0000 7f5aa718b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5aa00f6510 0x7f5aa01067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:50.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:50.920+0000 7f5aa718b700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa0106cd0 con 0x7f5aa00f6510 2024-02-19T22:18:50.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:50.920+0000 7f5aa4f27700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5aa00f6510 0x7f5aa01067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:50.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:50.920+0000 7f5aa4f27700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5aa00f6510 0x7f5aa01067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:53740/0 (socket says 172.21.15.88:53740) 2024-02-19T22:18:50.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:50.920+0000 7f5aa4f27700 1 -- 172.21.15.88:0/4080962479 learned_addr learned my addr 172.21.15.88:0/4080962479 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:18:50.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:50.920+0000 7f5aa4f27700 1 -- 172.21.15.88:0/4080962479 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a94005040 con 0x7f5aa00f6510 2024-02-19T22:18:50.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:50.920+0000 7f5aa4f27700 1 --2- 172.21.15.88:0/4080962479 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5aa00f6510 0x7f5aa01067e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f5a94005010 tx=0x7f5a9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:50.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:50.921+0000 7f5a9d7fa700 1 -- 172.21.15.88:0/4080962479 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a94009320 con 0x7f5aa00f6510 2024-02-19T22:18:50.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:50.921+0000 7f5a9d7fa700 1 -- 172.21.15.88:0/4080962479 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5a94013380 con 0x7f5aa00f6510 2024-02-19T22:18:50.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:50.921+0000 7f5a9d7fa700 1 -- 172.21.15.88:0/4080962479 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a94014e50 con 0x7f5aa00f6510 2024-02-19T22:18:50.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:50.921+0000 7f5aa718b700 1 -- 172.21.15.88:0/4080962479 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aa0106ed0 con 0x7f5aa00f6510 2024-02-19T22:18:50.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:50.921+0000 7f5aa718b700 1 -- 172.21.15.88:0/4080962479 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aa0107360 con 0x7f5aa00f6510 2024-02-19T22:18:50.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:50.921+0000 7f5a9d7fa700 1 -- 172.21.15.88:0/4080962479 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5a940095f0 con 0x7f5aa00f6510 2024-02-19T22:18:50.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:50.921+0000 7f5aa718b700 1 -- 172.21.15.88:0/4080962479 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5aa0045ad0 con 0x7f5aa00f6510 2024-02-19T22:18:50.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:50.922+0000 7f5a9d7fa700 1 --2- 172.21.15.88:0/4080962479 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f5a9002ff10 0x7f5a900323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:50.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:50.922+0000 7f5a9d7fa700 1 -- 172.21.15.88:0/4080962479 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5a94040f70 con 0x7f5aa00f6510 2024-02-19T22:18:50.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:50.922+0000 7f5a9ffff700 1 --2- 172.21.15.88:0/4080962479 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f5a9002ff10 0x7f5a900323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:50.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:50.922+0000 7f5a9ffff700 1 --2- 172.21.15.88:0/4080962479 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f5a9002ff10 0x7f5a900323d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f5a8c0096a0 tx=0x7f5a8c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:50.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:50.926+0000 7f5a9d7fa700 1 -- 172.21.15.88:0/4080962479 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5a94014420 con 0x7f5aa00f6510 2024-02-19T22:18:51.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:51.131+0000 7f5aa718b700 1 -- 172.21.15.88:0/4080962479 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5aa0108210 con 0x7f5aa00f6510 2024-02-19T22:18:51.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:51.132+0000 7f5a9d7fa700 1 -- 172.21.15.88:0/4080962479 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5a9401d980 con 0x7f5aa00f6510 2024-02-19T22:18:51.134 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:18:51.135 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:51.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:51.133+0000 7f5aa718b700 1 -- 172.21.15.88:0/4080962479 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f5a9002ff10 msgr2=0x7f5a900323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:51.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:51.133+0000 7f5aa718b700 1 --2- 172.21.15.88:0/4080962479 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f5a9002ff10 0x7f5a900323d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f5a8c0096a0 tx=0x7f5a8c006b40).stop 2024-02-19T22:18:51.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:51.133+0000 7f5aa718b700 1 -- 172.21.15.88:0/4080962479 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5aa00f6510 msgr2=0x7f5aa01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:51.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:51.133+0000 7f5aa718b700 1 --2- 172.21.15.88:0/4080962479 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5aa00f6510 0x7f5aa01067e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f5a94005010 tx=0x7f5a9400b040).stop 2024-02-19T22:18:51.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:51.134+0000 7f5aa718b700 1 -- 172.21.15.88:0/4080962479 shutdown_connections 2024-02-19T22:18:51.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:51.134+0000 7f5aa718b700 1 --2- 172.21.15.88:0/4080962479 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f5a9002ff10 0x7f5a900323d0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:51.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:51.134+0000 7f5aa718b700 1 --2- 172.21.15.88:0/4080962479 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5aa00f6510 0x7f5aa01067e0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:51.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:51.134+0000 7f5aa718b700 1 -- 172.21.15.88:0/4080962479 >> 172.21.15.88:0/4080962479 conn(0x7f5aa00f1c70 msgr2=0x7f5aa00f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:51.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:51.134+0000 7f5aa718b700 1 -- 172.21.15.88:0/4080962479 shutdown_connections 2024-02-19T22:18:51.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:51.134+0000 7f5aa718b700 1 -- 172.21.15.88:0/4080962479 wait complete. 2024-02-19T22:18:51.137 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:18:51.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:51 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:18:49.704466+0000 mgr.smithi052.ghqrtk (mgr.14162) 281 : cluster [DBG] pgmap v267: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:51.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:51 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:18:51.133434+0000 mon.smithi052 (mon.0) 281 : audit [DBG] from='client.? 172.21.15.88:0/4080962479' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:52.755 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:52.755 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:53.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:53 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:18:51.704848+0000 mgr.smithi052.ghqrtk (mgr.14162) 282 : cluster [DBG] pgmap v268: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:54.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:54.560+0000 7f759b0c8700 1 -- 172.21.15.88:0/212051883 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f75940f51a0 msgr2=0x7f75940f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:54.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:54.560+0000 7f759b0c8700 1 --2- 172.21.15.88:0/212051883 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f75940f51a0 0x7f75940f55c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f7584009fb0 tx=0x7f7584009b30).stop 2024-02-19T22:18:54.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:54.560+0000 7f759b0c8700 1 -- 172.21.15.88:0/212051883 shutdown_connections 2024-02-19T22:18:54.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:54.560+0000 7f759b0c8700 1 --2- 172.21.15.88:0/212051883 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f75940f51a0 0x7f75940f55c0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:54.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:54.560+0000 7f759b0c8700 1 -- 172.21.15.88:0/212051883 >> 172.21.15.88:0/212051883 conn(0x7f75940f09a0 msgr2=0x7f75940f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:54.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:54.560+0000 7f759b0c8700 1 -- 172.21.15.88:0/212051883 shutdown_connections 2024-02-19T22:18:54.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:54.560+0000 7f759b0c8700 1 -- 172.21.15.88:0/212051883 wait complete. 2024-02-19T22:18:54.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:54.560+0000 7f759b0c8700 1 Processor -- start 2024-02-19T22:18:54.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:54.560+0000 7f759b0c8700 1 -- start start 2024-02-19T22:18:54.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:54.560+0000 7f759b0c8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f75940f51a0 0x7f7594109290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:54.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:54.560+0000 7f759b0c8700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7594060280 con 0x7f75940f51a0 2024-02-19T22:18:54.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:54.561+0000 7f759a0c6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f75940f51a0 0x7f7594109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:54.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:54.561+0000 7f759a0c6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f75940f51a0 0x7f7594109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:38516/0 (socket says 172.21.15.88:38516) 2024-02-19T22:18:54.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:54.561+0000 7f759a0c6700 1 -- 172.21.15.88:0/1809569705 learned_addr learned my addr 172.21.15.88:0/1809569705 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:18:54.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:54.561+0000 7f759a0c6700 1 -- 172.21.15.88:0/1809569705 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7584005040 con 0x7f75940f51a0 2024-02-19T22:18:54.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:54.561+0000 7f759a0c6700 1 --2- 172.21.15.88:0/1809569705 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f75940f51a0 0x7f7594109290 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f75840089f0 tx=0x7f758400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:54.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:54.561+0000 7f758affd700 1 -- 172.21.15.88:0/1809569705 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7584009320 con 0x7f75940f51a0 2024-02-19T22:18:54.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:54.561+0000 7f759b0c8700 1 -- 172.21.15.88:0/1809569705 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7594060480 con 0x7f75940f51a0 2024-02-19T22:18:54.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:54.561+0000 7f758affd700 1 -- 172.21.15.88:0/1809569705 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7584013380 con 0x7f75940f51a0 2024-02-19T22:18:54.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:54.561+0000 7f758affd700 1 -- 172.21.15.88:0/1809569705 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7584014e20 con 0x7f75940f51a0 2024-02-19T22:18:54.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:54.561+0000 7f759b0c8700 1 -- 172.21.15.88:0/1809569705 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75941099c0 con 0x7f75940f51a0 2024-02-19T22:18:54.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:54.562+0000 7f758affd700 1 -- 172.21.15.88:0/1809569705 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f75840095f0 con 0x7f75940f51a0 2024-02-19T22:18:54.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:54.562+0000 7f759b0c8700 1 -- 172.21.15.88:0/1809569705 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7594045ad0 con 0x7f75940f51a0 2024-02-19T22:18:54.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:54.562+0000 7f758affd700 1 --2- 172.21.15.88:0/1809569705 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f758002ff10 0x7f75800323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:54.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:54.562+0000 7f758affd700 1 -- 172.21.15.88:0/1809569705 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7584040e90 con 0x7f75940f51a0 2024-02-19T22:18:54.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:54.562+0000 7f75998c5700 1 --2- 172.21.15.88:0/1809569705 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f758002ff10 0x7f75800323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:54.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:54.563+0000 7f75998c5700 1 --2- 172.21.15.88:0/1809569705 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f758002ff10 0x7f75800323d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f75900096a0 tx=0x7f7590006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:54.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:54.567+0000 7f758affd700 1 -- 172.21.15.88:0/1809569705 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7584014420 con 0x7f75940f51a0 2024-02-19T22:18:54.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:54.771+0000 7f759b0c8700 1 -- 172.21.15.88:0/1809569705 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f75940fad30 con 0x7f75940f51a0 2024-02-19T22:18:54.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:54.772+0000 7f758affd700 1 -- 172.21.15.88:0/1809569705 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f758401d980 con 0x7f75940f51a0 2024-02-19T22:18:54.775 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:18:54.775 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:54.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:54.774+0000 7f759b0c8700 1 -- 172.21.15.88:0/1809569705 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f758002ff10 msgr2=0x7f75800323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:54.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:54.774+0000 7f759b0c8700 1 --2- 172.21.15.88:0/1809569705 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f758002ff10 0x7f75800323d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f75900096a0 tx=0x7f7590006b40).stop 2024-02-19T22:18:54.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:54.774+0000 7f759b0c8700 1 -- 172.21.15.88:0/1809569705 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f75940f51a0 msgr2=0x7f7594109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:54.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:54.774+0000 7f759b0c8700 1 --2- 172.21.15.88:0/1809569705 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f75940f51a0 0x7f7594109290 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f75840089f0 tx=0x7f758400b040).stop 2024-02-19T22:18:54.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:54.774+0000 7f759b0c8700 1 -- 172.21.15.88:0/1809569705 shutdown_connections 2024-02-19T22:18:54.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:54.774+0000 7f759b0c8700 1 --2- 172.21.15.88:0/1809569705 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f758002ff10 0x7f75800323d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:54.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:54.774+0000 7f759b0c8700 1 --2- 172.21.15.88:0/1809569705 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f75940f51a0 0x7f7594109290 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:54.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:54.774+0000 7f759b0c8700 1 -- 172.21.15.88:0/1809569705 >> 172.21.15.88:0/1809569705 conn(0x7f75940f09a0 msgr2=0x7f75940f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:54.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:54.774+0000 7f759b0c8700 1 -- 172.21.15.88:0/1809569705 shutdown_connections 2024-02-19T22:18:54.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:54.774+0000 7f759b0c8700 1 -- 172.21.15.88:0/1809569705 wait complete. 2024-02-19T22:18:54.777 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:18:55.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:55 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:18:53.705207+0000 mgr.smithi052.ghqrtk (mgr.14162) 283 : cluster [DBG] pgmap v269: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:55.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:55 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:18:54.773500+0000 mon.smithi052 (mon.0) 282 : audit [DBG] from='client.? 172.21.15.88:0/1809569705' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:56.585 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:56.586 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:57.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:57 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:18:55.705567+0000 mgr.smithi052.ghqrtk (mgr.14162) 284 : cluster [DBG] pgmap v270: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:58.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:58.387+0000 7f905132b700 1 -- 172.21.15.88:0/1349919213 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f904c0f59d0 msgr2=0x7f904c0f5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:58.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:58.387+0000 7f905132b700 1 --2- 172.21.15.88:0/1349919213 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f904c0f59d0 0x7f904c0f5df0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f903c009fb0 tx=0x7f903c009b30).stop 2024-02-19T22:18:58.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:58.387+0000 7f905132b700 1 -- 172.21.15.88:0/1349919213 shutdown_connections 2024-02-19T22:18:58.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:58.387+0000 7f905132b700 1 --2- 172.21.15.88:0/1349919213 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f904c0f59d0 0x7f904c0f5df0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:58.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:58.387+0000 7f905132b700 1 -- 172.21.15.88:0/1349919213 >> 172.21.15.88:0/1349919213 conn(0x7f904c0f02e0 msgr2=0x7f904c0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:58.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:58.387+0000 7f905132b700 1 -- 172.21.15.88:0/1349919213 shutdown_connections 2024-02-19T22:18:58.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:58.387+0000 7f905132b700 1 -- 172.21.15.88:0/1349919213 wait complete. 2024-02-19T22:18:58.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:58.388+0000 7f905132b700 1 Processor -- start 2024-02-19T22:18:58.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:58.388+0000 7f905132b700 1 -- start start 2024-02-19T22:18:58.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:58.388+0000 7f905132b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f904c0f59d0 0x7f904c106790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:58.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:58.388+0000 7f905132b700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f904c106c80 con 0x7f904c0f59d0 2024-02-19T22:18:58.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:58.388+0000 7f904affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f904c0f59d0 0x7f904c106790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:58.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:58.388+0000 7f904affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f904c0f59d0 0x7f904c106790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:38532/0 (socket says 172.21.15.88:38532) 2024-02-19T22:18:58.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:58.388+0000 7f904affd700 1 -- 172.21.15.88:0/3528350446 learned_addr learned my addr 172.21.15.88:0/3528350446 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:18:58.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:58.389+0000 7f904affd700 1 -- 172.21.15.88:0/3528350446 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f903c005040 con 0x7f904c0f59d0 2024-02-19T22:18:58.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:58.389+0000 7f904affd700 1 --2- 172.21.15.88:0/3528350446 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f904c0f59d0 0x7f904c106790 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f903c005010 tx=0x7f903c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:58.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:58.389+0000 7f90337fe700 1 -- 172.21.15.88:0/3528350446 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f903c009320 con 0x7f904c0f59d0 2024-02-19T22:18:58.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:58.389+0000 7f905132b700 1 -- 172.21.15.88:0/3528350446 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f904c106e80 con 0x7f904c0f59d0 2024-02-19T22:18:58.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:58.389+0000 7f90337fe700 1 -- 172.21.15.88:0/3528350446 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f903c013380 con 0x7f904c0f59d0 2024-02-19T22:18:58.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:58.389+0000 7f90337fe700 1 -- 172.21.15.88:0/3528350446 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f903c014e50 con 0x7f904c0f59d0 2024-02-19T22:18:58.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:58.389+0000 7f905132b700 1 -- 172.21.15.88:0/3528350446 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f904c107310 con 0x7f904c0f59d0 2024-02-19T22:18:58.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:58.390+0000 7f90337fe700 1 -- 172.21.15.88:0/3528350446 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f903c0095f0 con 0x7f904c0f59d0 2024-02-19T22:18:58.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:58.390+0000 7f905132b700 1 -- 172.21.15.88:0/3528350446 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f904c045ad0 con 0x7f904c0f59d0 2024-02-19T22:18:58.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:58.390+0000 7f90337fe700 1 --2- 172.21.15.88:0/3528350446 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f903402ff10 0x7f90340323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:58.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:58.390+0000 7f90337fe700 1 -- 172.21.15.88:0/3528350446 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f903c040e90 con 0x7f904c0f59d0 2024-02-19T22:18:58.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:58.390+0000 7f904a7fc700 1 --2- 172.21.15.88:0/3528350446 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f903402ff10 0x7f90340323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:58.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:58.390+0000 7f904a7fc700 1 --2- 172.21.15.88:0/3528350446 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f903402ff10 0x7f90340323d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f90400096a0 tx=0x7f9040006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:58.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:58.394+0000 7f90337fe700 1 -- 172.21.15.88:0/3528350446 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f903c014420 con 0x7f904c0f59d0 2024-02-19T22:18:58.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:58.599+0000 7f905132b700 1 -- 172.21.15.88:0/3528350446 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f904c1081c0 con 0x7f904c0f59d0 2024-02-19T22:18:58.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:58.600+0000 7f90337fe700 1 -- 172.21.15.88:0/3528350446 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f903c010230 con 0x7f904c0f59d0 2024-02-19T22:18:58.603 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:18:58.603 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:58.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:58.601+0000 7f905132b700 1 -- 172.21.15.88:0/3528350446 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f903402ff10 msgr2=0x7f90340323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:58.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:58.601+0000 7f905132b700 1 --2- 172.21.15.88:0/3528350446 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f903402ff10 0x7f90340323d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f90400096a0 tx=0x7f9040006b40).stop 2024-02-19T22:18:58.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:58.601+0000 7f905132b700 1 -- 172.21.15.88:0/3528350446 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f904c0f59d0 msgr2=0x7f904c106790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:58.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:58.601+0000 7f905132b700 1 --2- 172.21.15.88:0/3528350446 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f904c0f59d0 0x7f904c106790 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f903c005010 tx=0x7f903c00b040).stop 2024-02-19T22:18:58.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:58.602+0000 7f905132b700 1 -- 172.21.15.88:0/3528350446 shutdown_connections 2024-02-19T22:18:58.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:58.602+0000 7f905132b700 1 --2- 172.21.15.88:0/3528350446 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f903402ff10 0x7f90340323d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:58.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:58.602+0000 7f905132b700 1 --2- 172.21.15.88:0/3528350446 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f904c0f59d0 0x7f904c106790 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:58.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:58.602+0000 7f905132b700 1 -- 172.21.15.88:0/3528350446 >> 172.21.15.88:0/3528350446 conn(0x7f904c0f02e0 msgr2=0x7f904c0f4f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:58.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:58.602+0000 7f905132b700 1 -- 172.21.15.88:0/3528350446 shutdown_connections 2024-02-19T22:18:58.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:18:58.602+0000 7f905132b700 1 -- 172.21.15.88:0/3528350446 wait complete. 2024-02-19T22:18:58.605 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:18:59.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:59 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:18:57.705829+0000 mgr.smithi052.ghqrtk (mgr.14162) 285 : cluster [DBG] pgmap v271: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:59.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:18:59 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:18:58.601548+0000 mon.smithi052 (mon.0) 283 : audit [DBG] from='client.? 172.21.15.88:0/3528350446' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:00.375 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:00.376 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:01.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:01 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:18:59.706183+0000 mgr.smithi052.ghqrtk (mgr.14162) 286 : cluster [DBG] pgmap v272: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:02.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:02.213+0000 7f45ec243700 1 -- 172.21.15.88:0/77193586 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f45e40f5410 msgr2=0x7f45e40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:02.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:02.213+0000 7f45ec243700 1 --2- 172.21.15.88:0/77193586 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f45e40f5410 0x7f45e40f5830 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f45e0009fb0 tx=0x7f45e0009b30).stop 2024-02-19T22:19:02.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:02.213+0000 7f45ec243700 1 -- 172.21.15.88:0/77193586 shutdown_connections 2024-02-19T22:19:02.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:02.213+0000 7f45ec243700 1 --2- 172.21.15.88:0/77193586 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f45e40f5410 0x7f45e40f5830 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:02.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:02.213+0000 7f45ec243700 1 -- 172.21.15.88:0/77193586 >> 172.21.15.88:0/77193586 conn(0x7f45e40f0b70 msgr2=0x7f45e40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:02.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:02.213+0000 7f45ec243700 1 -- 172.21.15.88:0/77193586 shutdown_connections 2024-02-19T22:19:02.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:02.213+0000 7f45ec243700 1 -- 172.21.15.88:0/77193586 wait complete. 2024-02-19T22:19:02.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:02.213+0000 7f45ec243700 1 Processor -- start 2024-02-19T22:19:02.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:02.213+0000 7f45ec243700 1 -- start start 2024-02-19T22:19:02.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:02.213+0000 7f45ec243700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f45e40f5410 0x7f45e41067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:02.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:02.213+0000 7f45ec243700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45e4106cd0 con 0x7f45e40f5410 2024-02-19T22:19:02.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:02.214+0000 7f45e9fdf700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f45e40f5410 0x7f45e41067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:02.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:02.214+0000 7f45e9fdf700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f45e40f5410 0x7f45e41067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:38560/0 (socket says 172.21.15.88:38560) 2024-02-19T22:19:02.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:02.214+0000 7f45e9fdf700 1 -- 172.21.15.88:0/3956621584 learned_addr learned my addr 172.21.15.88:0/3956621584 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:19:02.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:02.214+0000 7f45e9fdf700 1 -- 172.21.15.88:0/3956621584 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45e0005040 con 0x7f45e40f5410 2024-02-19T22:19:02.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:02.214+0000 7f45e9fdf700 1 --2- 172.21.15.88:0/3956621584 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f45e40f5410 0x7f45e41067e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f45e0005010 tx=0x7f45e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:02.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:02.214+0000 7f45da7fc700 1 -- 172.21.15.88:0/3956621584 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45e0009320 con 0x7f45e40f5410 2024-02-19T22:19:02.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:02.214+0000 7f45ec243700 1 -- 172.21.15.88:0/3956621584 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45e4106ed0 con 0x7f45e40f5410 2024-02-19T22:19:02.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:02.214+0000 7f45da7fc700 1 -- 172.21.15.88:0/3956621584 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f45e0013380 con 0x7f45e40f5410 2024-02-19T22:19:02.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:02.214+0000 7f45da7fc700 1 -- 172.21.15.88:0/3956621584 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45e0014e50 con 0x7f45e40f5410 2024-02-19T22:19:02.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:02.214+0000 7f45ec243700 1 -- 172.21.15.88:0/3956621584 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45e4107360 con 0x7f45e40f5410 2024-02-19T22:19:02.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:02.215+0000 7f45da7fc700 1 -- 172.21.15.88:0/3956621584 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f45e00095f0 con 0x7f45e40f5410 2024-02-19T22:19:02.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:02.215+0000 7f45ec243700 1 -- 172.21.15.88:0/3956621584 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45e4045ad0 con 0x7f45e40f5410 2024-02-19T22:19:02.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:02.215+0000 7f45da7fc700 1 --2- 172.21.15.88:0/3956621584 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f45d002ff10 0x7f45d00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:02.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:02.215+0000 7f45da7fc700 1 -- 172.21.15.88:0/3956621584 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f45e0040f60 con 0x7f45e40f5410 2024-02-19T22:19:02.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:02.215+0000 7f45e97de700 1 --2- 172.21.15.88:0/3956621584 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f45d002ff10 0x7f45d00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:02.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:02.216+0000 7f45e97de700 1 --2- 172.21.15.88:0/3956621584 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f45d002ff10 0x7f45d00323d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f45d40096a0 tx=0x7f45d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:02.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:02.220+0000 7f45da7fc700 1 -- 172.21.15.88:0/3956621584 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f45e0014420 con 0x7f45e40f5410 2024-02-19T22:19:02.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:02.425+0000 7f45ec243700 1 -- 172.21.15.88:0/3956621584 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f45e40fafa0 con 0x7f45e40f5410 2024-02-19T22:19:02.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:02.425+0000 7f45da7fc700 1 -- 172.21.15.88:0/3956621584 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f45e0010230 con 0x7f45e40f5410 2024-02-19T22:19:02.428 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:19:02.428 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:02.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:02.427+0000 7f45ec243700 1 -- 172.21.15.88:0/3956621584 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f45d002ff10 msgr2=0x7f45d00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:02.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:02.427+0000 7f45ec243700 1 --2- 172.21.15.88:0/3956621584 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f45d002ff10 0x7f45d00323d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f45d40096a0 tx=0x7f45d4006b40).stop 2024-02-19T22:19:02.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:02.427+0000 7f45ec243700 1 -- 172.21.15.88:0/3956621584 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f45e40f5410 msgr2=0x7f45e41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:02.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:02.427+0000 7f45ec243700 1 --2- 172.21.15.88:0/3956621584 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f45e40f5410 0x7f45e41067e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f45e0005010 tx=0x7f45e000b040).stop 2024-02-19T22:19:02.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:02.427+0000 7f45ec243700 1 -- 172.21.15.88:0/3956621584 shutdown_connections 2024-02-19T22:19:02.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:02.427+0000 7f45ec243700 1 --2- 172.21.15.88:0/3956621584 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f45d002ff10 0x7f45d00323d0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:02.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:02.427+0000 7f45ec243700 1 --2- 172.21.15.88:0/3956621584 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f45e40f5410 0x7f45e41067e0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:02.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:02.427+0000 7f45ec243700 1 -- 172.21.15.88:0/3956621584 >> 172.21.15.88:0/3956621584 conn(0x7f45e40f0b70 msgr2=0x7f45e405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:02.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:02.427+0000 7f45ec243700 1 -- 172.21.15.88:0/3956621584 shutdown_connections 2024-02-19T22:19:02.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:02.427+0000 7f45ec243700 1 -- 172.21.15.88:0/3956621584 wait complete. 2024-02-19T22:19:02.431 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:19:03.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:03 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:19:01.706544+0000 mgr.smithi052.ghqrtk (mgr.14162) 287 : cluster [DBG] pgmap v273: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:03.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:03 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:19:02.426736+0000 mon.smithi052 (mon.0) 284 : audit [DBG] from='client.? 172.21.15.88:0/3956621584' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:04.133 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:04.134 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:05.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:05 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:19:03.706896+0000 mgr.smithi052.ghqrtk (mgr.14162) 288 : cluster [DBG] pgmap v274: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:06.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:06.113+0000 7f81f2e9e700 1 -- 172.21.15.88:0/1231265214 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f81ec0f5410 msgr2=0x7f81ec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:06.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:06.113+0000 7f81f2e9e700 1 --2- 172.21.15.88:0/1231265214 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f81ec0f5410 0x7f81ec0f5830 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f81e0009fb0 tx=0x7f81e0009b30).stop 2024-02-19T22:19:06.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:06.113+0000 7f81f2e9e700 1 -- 172.21.15.88:0/1231265214 shutdown_connections 2024-02-19T22:19:06.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:06.113+0000 7f81f2e9e700 1 --2- 172.21.15.88:0/1231265214 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f81ec0f5410 0x7f81ec0f5830 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:06.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:06.113+0000 7f81f2e9e700 1 -- 172.21.15.88:0/1231265214 >> 172.21.15.88:0/1231265214 conn(0x7f81ec0f0b70 msgr2=0x7f81ec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:06.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:06.113+0000 7f81f2e9e700 1 -- 172.21.15.88:0/1231265214 shutdown_connections 2024-02-19T22:19:06.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:06.113+0000 7f81f2e9e700 1 -- 172.21.15.88:0/1231265214 wait complete. 2024-02-19T22:19:06.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:06.113+0000 7f81f2e9e700 1 Processor -- start 2024-02-19T22:19:06.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:06.114+0000 7f81f2e9e700 1 -- start start 2024-02-19T22:19:06.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:06.114+0000 7f81f2e9e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f81ec0f5410 0x7f81ec0fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:06.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:06.114+0000 7f81f2e9e700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81ec0fb960 con 0x7f81ec0f5410 2024-02-19T22:19:06.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:06.114+0000 7f81f0c3a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f81ec0f5410 0x7f81ec0fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:06.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:06.114+0000 7f81f0c3a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f81ec0f5410 0x7f81ec0fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:45986/0 (socket says 172.21.15.88:45986) 2024-02-19T22:19:06.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:06.114+0000 7f81f0c3a700 1 -- 172.21.15.88:0/3702355719 learned_addr learned my addr 172.21.15.88:0/3702355719 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:19:06.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:06.114+0000 7f81f0c3a700 1 -- 172.21.15.88:0/3702355719 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81e0005040 con 0x7f81ec0f5410 2024-02-19T22:19:06.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:06.114+0000 7f81f0c3a700 1 --2- 172.21.15.88:0/3702355719 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f81ec0f5410 0x7f81ec0fd4e0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f81e0005010 tx=0x7f81e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:06.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:06.115+0000 7f81e97fa700 1 -- 172.21.15.88:0/3702355719 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81e0009320 con 0x7f81ec0f5410 2024-02-19T22:19:06.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:06.115+0000 7f81f2e9e700 1 -- 172.21.15.88:0/3702355719 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81ec0fbb00 con 0x7f81ec0f5410 2024-02-19T22:19:06.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:06.115+0000 7f81e97fa700 1 -- 172.21.15.88:0/3702355719 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f81e0013380 con 0x7f81ec0f5410 2024-02-19T22:19:06.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:06.115+0000 7f81e97fa700 1 -- 172.21.15.88:0/3702355719 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81e0014e50 con 0x7f81ec0f5410 2024-02-19T22:19:06.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:06.115+0000 7f81f2e9e700 1 -- 172.21.15.88:0/3702355719 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81ec0fbf90 con 0x7f81ec0f5410 2024-02-19T22:19:06.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:06.115+0000 7f81e97fa700 1 -- 172.21.15.88:0/3702355719 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f81e0014420 con 0x7f81ec0f5410 2024-02-19T22:19:06.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:06.116+0000 7f81f2e9e700 1 -- 172.21.15.88:0/3702355719 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81ec059100 con 0x7f81ec0f5410 2024-02-19T22:19:06.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:06.116+0000 7f81e97fa700 1 --2- 172.21.15.88:0/3702355719 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f81dc02ff10 0x7f81dc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:06.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:06.116+0000 7f81e97fa700 1 -- 172.21.15.88:0/3702355719 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f81e0040eb0 con 0x7f81ec0f5410 2024-02-19T22:19:06.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:06.116+0000 7f81ebfff700 1 --2- 172.21.15.88:0/3702355719 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f81dc02ff10 0x7f81dc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:06.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:06.116+0000 7f81ebfff700 1 --2- 172.21.15.88:0/3702355719 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f81dc02ff10 0x7f81dc0323d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f81d80096a0 tx=0x7f81d8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:06.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:06.120+0000 7f81e97fa700 1 -- 172.21.15.88:0/3702355719 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f81e00095f0 con 0x7f81ec0f5410 2024-02-19T22:19:06.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:06.325+0000 7f81f2e9e700 1 -- 172.21.15.88:0/3702355719 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f81ec0fafa0 con 0x7f81ec0f5410 2024-02-19T22:19:06.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:06.326+0000 7f81e97fa700 1 -- 172.21.15.88:0/3702355719 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f81e001db70 con 0x7f81ec0f5410 2024-02-19T22:19:06.329 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:19:06.329 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:06.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:06.327+0000 7f81f2e9e700 1 -- 172.21.15.88:0/3702355719 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f81dc02ff10 msgr2=0x7f81dc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:06.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:06.327+0000 7f81f2e9e700 1 --2- 172.21.15.88:0/3702355719 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f81dc02ff10 0x7f81dc0323d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f81d80096a0 tx=0x7f81d8006b40).stop 2024-02-19T22:19:06.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:06.327+0000 7f81f2e9e700 1 -- 172.21.15.88:0/3702355719 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f81ec0f5410 msgr2=0x7f81ec0fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:06.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:06.327+0000 7f81f2e9e700 1 --2- 172.21.15.88:0/3702355719 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f81ec0f5410 0x7f81ec0fd4e0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f81e0005010 tx=0x7f81e000b040).stop 2024-02-19T22:19:06.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:06.327+0000 7f81f2e9e700 1 -- 172.21.15.88:0/3702355719 shutdown_connections 2024-02-19T22:19:06.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:06.327+0000 7f81f2e9e700 1 --2- 172.21.15.88:0/3702355719 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f81dc02ff10 0x7f81dc0323d0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:06.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:06.327+0000 7f81f2e9e700 1 --2- 172.21.15.88:0/3702355719 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f81ec0f5410 0x7f81ec0fd4e0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:06.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:06.327+0000 7f81f2e9e700 1 -- 172.21.15.88:0/3702355719 >> 172.21.15.88:0/3702355719 conn(0x7f81ec0f0b70 msgr2=0x7f81ec05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:06.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:06.327+0000 7f81f2e9e700 1 -- 172.21.15.88:0/3702355719 shutdown_connections 2024-02-19T22:19:06.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:06.327+0000 7f81f2e9e700 1 -- 172.21.15.88:0/3702355719 wait complete. 2024-02-19T22:19:06.331 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:19:07.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:07 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:19:05.707361+0000 mgr.smithi052.ghqrtk (mgr.14162) 289 : cluster [DBG] pgmap v275: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:07.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:07 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:19:06.327385+0000 mon.smithi052 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.88:0/3702355719' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:07.986 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:07.987 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:09.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:09 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:19:07.707624+0000 mgr.smithi052.ghqrtk (mgr.14162) 290 : cluster [DBG] pgmap v276: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:09.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:09.917+0000 7f5d6fc0a700 1 -- 172.21.15.88:0/1612114253 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5d680f5410 msgr2=0x7f5d680f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:09.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:09.917+0000 7f5d6fc0a700 1 --2- 172.21.15.88:0/1612114253 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5d680f5410 0x7f5d680f5830 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f5d58009fb0 tx=0x7f5d58009b30).stop 2024-02-19T22:19:09.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:09.917+0000 7f5d6fc0a700 1 -- 172.21.15.88:0/1612114253 shutdown_connections 2024-02-19T22:19:09.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:09.917+0000 7f5d6fc0a700 1 --2- 172.21.15.88:0/1612114253 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5d680f5410 0x7f5d680f5830 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:09.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:09.917+0000 7f5d6fc0a700 1 -- 172.21.15.88:0/1612114253 >> 172.21.15.88:0/1612114253 conn(0x7f5d680f0b70 msgr2=0x7f5d680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:09.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:09.917+0000 7f5d6fc0a700 1 -- 172.21.15.88:0/1612114253 shutdown_connections 2024-02-19T22:19:09.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:09.917+0000 7f5d6fc0a700 1 -- 172.21.15.88:0/1612114253 wait complete. 2024-02-19T22:19:09.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:09.917+0000 7f5d6fc0a700 1 Processor -- start 2024-02-19T22:19:09.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:09.918+0000 7f5d6fc0a700 1 -- start start 2024-02-19T22:19:09.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:09.918+0000 7f5d6fc0a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5d68060200 0x7f5d68109640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:09.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:09.918+0000 7f5d6fc0a700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d68109b30 con 0x7f5d68060200 2024-02-19T22:19:09.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:09.918+0000 7f5d6d9a6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5d68060200 0x7f5d68109640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:09.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:09.918+0000 7f5d6d9a6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5d68060200 0x7f5d68109640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:46006/0 (socket says 172.21.15.88:46006) 2024-02-19T22:19:09.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:09.918+0000 7f5d6d9a6700 1 -- 172.21.15.88:0/2714490329 learned_addr learned my addr 172.21.15.88:0/2714490329 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:19:09.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:09.918+0000 7f5d6d9a6700 1 -- 172.21.15.88:0/2714490329 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d58005040 con 0x7f5d68060200 2024-02-19T22:19:09.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:09.919+0000 7f5d6d9a6700 1 --2- 172.21.15.88:0/2714490329 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5d68060200 0x7f5d68109640 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f5d680f6510 tx=0x7f5d58009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:09.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:09.919+0000 7f5d5e7fc700 1 -- 172.21.15.88:0/2714490329 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d58013380 con 0x7f5d68060200 2024-02-19T22:19:09.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:09.919+0000 7f5d6fc0a700 1 -- 172.21.15.88:0/2714490329 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d68109c70 con 0x7f5d68060200 2024-02-19T22:19:09.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:09.919+0000 7f5d5e7fc700 1 -- 172.21.15.88:0/2714490329 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5d58013790 con 0x7f5d68060200 2024-02-19T22:19:09.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:09.919+0000 7f5d5e7fc700 1 -- 172.21.15.88:0/2714490329 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d5801ee70 con 0x7f5d68060200 2024-02-19T22:19:09.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:09.919+0000 7f5d6fc0a700 1 -- 172.21.15.88:0/2714490329 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d6810a0a0 con 0x7f5d68060200 2024-02-19T22:19:09.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:09.919+0000 7f5d5e7fc700 1 -- 172.21.15.88:0/2714490329 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5d58025060 con 0x7f5d68060200 2024-02-19T22:19:09.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:09.920+0000 7f5d5e7fc700 1 --2- 172.21.15.88:0/2714490329 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f5d5402ff60 0x7f5d54032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:09.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:09.920+0000 7f5d5e7fc700 1 -- 172.21.15.88:0/2714490329 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5d58040dd0 con 0x7f5d68060200 2024-02-19T22:19:09.924 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:09.920+0000 7f5d6fc0a700 1 -- 172.21.15.88:0/2714490329 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d68045ad0 con 0x7f5d68060200 2024-02-19T22:19:09.924 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:09.920+0000 7f5d6d1a5700 1 --2- 172.21.15.88:0/2714490329 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f5d5402ff60 0x7f5d54032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:09.924 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:09.920+0000 7f5d6d1a5700 1 --2- 172.21.15.88:0/2714490329 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f5d5402ff60 0x7f5d54032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f5d640096a0 tx=0x7f5d64006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:09.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:09.925+0000 7f5d5e7fc700 1 -- 172.21.15.88:0/2714490329 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5d58028440 con 0x7f5d68060200 2024-02-19T22:19:10.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:10.130+0000 7f5d6fc0a700 1 -- 172.21.15.88:0/2714490329 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5d6810a8c0 con 0x7f5d68060200 2024-02-19T22:19:10.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:10.131+0000 7f5d5e7fc700 1 -- 172.21.15.88:0/2714490329 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5d5801d3a0 con 0x7f5d68060200 2024-02-19T22:19:10.134 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:19:10.134 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:10.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:10.133+0000 7f5d6fc0a700 1 -- 172.21.15.88:0/2714490329 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f5d5402ff60 msgr2=0x7f5d54032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:10.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:10.133+0000 7f5d6fc0a700 1 --2- 172.21.15.88:0/2714490329 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f5d5402ff60 0x7f5d54032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f5d640096a0 tx=0x7f5d64006b40).stop 2024-02-19T22:19:10.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:10.133+0000 7f5d6fc0a700 1 -- 172.21.15.88:0/2714490329 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5d68060200 msgr2=0x7f5d68109640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:10.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:10.133+0000 7f5d6fc0a700 1 --2- 172.21.15.88:0/2714490329 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5d68060200 0x7f5d68109640 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f5d680f6510 tx=0x7f5d58009b30).stop 2024-02-19T22:19:10.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:10.133+0000 7f5d6fc0a700 1 -- 172.21.15.88:0/2714490329 shutdown_connections 2024-02-19T22:19:10.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:10.133+0000 7f5d6fc0a700 1 --2- 172.21.15.88:0/2714490329 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f5d5402ff60 0x7f5d54032420 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:10.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:10.133+0000 7f5d6fc0a700 1 --2- 172.21.15.88:0/2714490329 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5d68060200 0x7f5d68109640 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:10.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:10.133+0000 7f5d6fc0a700 1 -- 172.21.15.88:0/2714490329 >> 172.21.15.88:0/2714490329 conn(0x7f5d680f0b70 msgr2=0x7f5d680f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:10.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:10.133+0000 7f5d6fc0a700 1 -- 172.21.15.88:0/2714490329 shutdown_connections 2024-02-19T22:19:10.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:10.133+0000 7f5d6fc0a700 1 -- 172.21.15.88:0/2714490329 wait complete. 2024-02-19T22:19:10.136 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:19:10.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:10 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:19:10.132207+0000 mon.smithi052 (mon.0) 286 : audit [DBG] from='client.? 172.21.15.88:0/2714490329' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:11.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:11 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:19:09.707838+0000 mgr.smithi052.ghqrtk (mgr.14162) 291 : cluster [DBG] pgmap v277: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:11.791 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:11.792 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:13.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:13.587+0000 7fd9c47de700 1 -- 172.21.15.88:0/2088843702 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd9bc0f5410 msgr2=0x7fd9bc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:13.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:13.587+0000 7fd9c47de700 1 --2- 172.21.15.88:0/2088843702 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd9bc0f5410 0x7fd9bc0f5830 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7fd9ac009fb0 tx=0x7fd9ac009b30).stop 2024-02-19T22:19:13.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:13.587+0000 7fd9c47de700 1 -- 172.21.15.88:0/2088843702 shutdown_connections 2024-02-19T22:19:13.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:13.587+0000 7fd9c47de700 1 --2- 172.21.15.88:0/2088843702 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd9bc0f5410 0x7fd9bc0f5830 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:13.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:13.587+0000 7fd9c47de700 1 -- 172.21.15.88:0/2088843702 >> 172.21.15.88:0/2088843702 conn(0x7fd9bc0f0b70 msgr2=0x7fd9bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:13.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:13.587+0000 7fd9c47de700 1 -- 172.21.15.88:0/2088843702 shutdown_connections 2024-02-19T22:19:13.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:13.587+0000 7fd9c47de700 1 -- 172.21.15.88:0/2088843702 wait complete. 2024-02-19T22:19:13.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:13.588+0000 7fd9c47de700 1 Processor -- start 2024-02-19T22:19:13.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:13.588+0000 7fd9c47de700 1 -- start start 2024-02-19T22:19:13.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:13.588+0000 7fd9c47de700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd9bc0f5410 0x7fd9bc1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:13.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:13.588+0000 7fd9c47de700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9bc105bd0 con 0x7fd9bc0f5410 2024-02-19T22:19:13.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:13.588+0000 7fd9c257a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd9bc0f5410 0x7fd9bc1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:13.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:13.588+0000 7fd9c257a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd9bc0f5410 0x7fd9bc1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:49144/0 (socket says 172.21.15.88:49144) 2024-02-19T22:19:13.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:13.588+0000 7fd9c257a700 1 -- 172.21.15.88:0/4050801553 learned_addr learned my addr 172.21.15.88:0/4050801553 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:19:13.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:13.589+0000 7fd9c257a700 1 -- 172.21.15.88:0/4050801553 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9ac005040 con 0x7fd9bc0f5410 2024-02-19T22:19:13.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:13.589+0000 7fd9c257a700 1 --2- 172.21.15.88:0/4050801553 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd9bc0f5410 0x7fd9bc1056e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7fd9ac005010 tx=0x7fd9ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:13.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:13.589+0000 7fd9b2ffd700 1 -- 172.21.15.88:0/4050801553 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd9ac009320 con 0x7fd9bc0f5410 2024-02-19T22:19:13.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:13.589+0000 7fd9b2ffd700 1 -- 172.21.15.88:0/4050801553 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd9ac013380 con 0x7fd9bc0f5410 2024-02-19T22:19:13.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:13.589+0000 7fd9c47de700 1 -- 172.21.15.88:0/4050801553 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9bc105dd0 con 0x7fd9bc0f5410 2024-02-19T22:19:13.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:13.589+0000 7fd9b2ffd700 1 -- 172.21.15.88:0/4050801553 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd9ac014e50 con 0x7fd9bc0f5410 2024-02-19T22:19:13.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:13.589+0000 7fd9c47de700 1 -- 172.21.15.88:0/4050801553 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9bc106260 con 0x7fd9bc0f5410 2024-02-19T22:19:13.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:13 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:19:11.708137+0000 mgr.smithi052.ghqrtk (mgr.14162) 292 : cluster [DBG] pgmap v278: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:13.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:13.590+0000 7fd9b2ffd700 1 -- 172.21.15.88:0/4050801553 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd9ac0095f0 con 0x7fd9bc0f5410 2024-02-19T22:19:13.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:13.590+0000 7fd9c47de700 1 -- 172.21.15.88:0/4050801553 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9bc045ad0 con 0x7fd9bc0f5410 2024-02-19T22:19:13.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:13.590+0000 7fd9b2ffd700 1 --2- 172.21.15.88:0/4050801553 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd9a802ff10 0x7fd9a80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:13.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:13.590+0000 7fd9b2ffd700 1 -- 172.21.15.88:0/4050801553 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd9ac040ef0 con 0x7fd9bc0f5410 2024-02-19T22:19:13.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:13.590+0000 7fd9c1d79700 1 --2- 172.21.15.88:0/4050801553 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd9a802ff10 0x7fd9a80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:13.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:13.591+0000 7fd9c1d79700 1 --2- 172.21.15.88:0/4050801553 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd9a802ff10 0x7fd9a80323d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fd9b80096a0 tx=0x7fd9b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:13.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:13.595+0000 7fd9b2ffd700 1 -- 172.21.15.88:0/4050801553 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd9ac014420 con 0x7fd9bc0f5410 2024-02-19T22:19:13.799 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:13.797+0000 7fd9c47de700 1 -- 172.21.15.88:0/4050801553 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd9bc107110 con 0x7fd9bc0f5410 2024-02-19T22:19:13.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:13.798+0000 7fd9b2ffd700 1 -- 172.21.15.88:0/4050801553 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd9ac010230 con 0x7fd9bc0f5410 2024-02-19T22:19:13.800 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:19:13.800 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:13.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:13.799+0000 7fd9c47de700 1 -- 172.21.15.88:0/4050801553 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd9a802ff10 msgr2=0x7fd9a80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:13.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:13.799+0000 7fd9c47de700 1 --2- 172.21.15.88:0/4050801553 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd9a802ff10 0x7fd9a80323d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fd9b80096a0 tx=0x7fd9b8006b40).stop 2024-02-19T22:19:13.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:13.799+0000 7fd9c47de700 1 -- 172.21.15.88:0/4050801553 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd9bc0f5410 msgr2=0x7fd9bc1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:13.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:13.799+0000 7fd9c47de700 1 --2- 172.21.15.88:0/4050801553 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd9bc0f5410 0x7fd9bc1056e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7fd9ac005010 tx=0x7fd9ac00b040).stop 2024-02-19T22:19:13.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:13.799+0000 7fd9c47de700 1 -- 172.21.15.88:0/4050801553 shutdown_connections 2024-02-19T22:19:13.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:13.799+0000 7fd9c47de700 1 --2- 172.21.15.88:0/4050801553 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd9a802ff10 0x7fd9a80323d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:13.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:13.799+0000 7fd9c47de700 1 --2- 172.21.15.88:0/4050801553 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd9bc0f5410 0x7fd9bc1056e0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:13.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:13.799+0000 7fd9c47de700 1 -- 172.21.15.88:0/4050801553 >> 172.21.15.88:0/4050801553 conn(0x7fd9bc0f0b70 msgr2=0x7fd9bc0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:13.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:13.799+0000 7fd9c47de700 1 -- 172.21.15.88:0/4050801553 shutdown_connections 2024-02-19T22:19:13.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:13.799+0000 7fd9c47de700 1 -- 172.21.15.88:0/4050801553 wait complete. 2024-02-19T22:19:13.803 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:19:14.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:14 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:19:13.798942+0000 mon.smithi052 (mon.0) 287 : audit [DBG] from='client.? 172.21.15.88:0/4050801553' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:15.550 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:15.550 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:15.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:15 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:19:13.708475+0000 mgr.smithi052.ghqrtk (mgr.14162) 293 : cluster [DBG] pgmap v279: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:17.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:17.391+0000 7f87b6eb9700 1 -- 172.21.15.88:0/3285415798 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f87b00f5410 msgr2=0x7f87b00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:17.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:17.391+0000 7f87b6eb9700 1 --2- 172.21.15.88:0/3285415798 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f87b00f5410 0x7f87b00f5830 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f87a4009fb0 tx=0x7f87a4009b30).stop 2024-02-19T22:19:17.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:17.391+0000 7f87b6eb9700 1 -- 172.21.15.88:0/3285415798 shutdown_connections 2024-02-19T22:19:17.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:17.391+0000 7f87b6eb9700 1 --2- 172.21.15.88:0/3285415798 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f87b00f5410 0x7f87b00f5830 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:17.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:17.391+0000 7f87b6eb9700 1 -- 172.21.15.88:0/3285415798 >> 172.21.15.88:0/3285415798 conn(0x7f87b00f0b70 msgr2=0x7f87b00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:17.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:17.391+0000 7f87b6eb9700 1 -- 172.21.15.88:0/3285415798 shutdown_connections 2024-02-19T22:19:17.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:17.391+0000 7f87b6eb9700 1 -- 172.21.15.88:0/3285415798 wait complete. 2024-02-19T22:19:17.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:17.391+0000 7f87b6eb9700 1 Processor -- start 2024-02-19T22:19:17.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:17.391+0000 7f87b6eb9700 1 -- start start 2024-02-19T22:19:17.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:17.391+0000 7f87b6eb9700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f87b00f5410 0x7f87b010b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:17.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:17.391+0000 7f87b6eb9700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b01072f0 con 0x7f87b00f5410 2024-02-19T22:19:17.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:17.392+0000 7f87b4c55700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f87b00f5410 0x7f87b010b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:17.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:17.392+0000 7f87b4c55700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f87b00f5410 0x7f87b010b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:49160/0 (socket says 172.21.15.88:49160) 2024-02-19T22:19:17.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:17.392+0000 7f87b4c55700 1 -- 172.21.15.88:0/78144752 learned_addr learned my addr 172.21.15.88:0/78144752 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:19:17.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:17.392+0000 7f87b4c55700 1 -- 172.21.15.88:0/78144752 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87a4005040 con 0x7f87b00f5410 2024-02-19T22:19:17.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:17.392+0000 7f87b4c55700 1 --2- 172.21.15.88:0/78144752 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f87b00f5410 0x7f87b010b900 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f87a40089f0 tx=0x7f87a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:17.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:17.392+0000 7f87ad7fa700 1 -- 172.21.15.88:0/78144752 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87a4009320 con 0x7f87b00f5410 2024-02-19T22:19:17.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:17.392+0000 7f87b6eb9700 1 -- 172.21.15.88:0/78144752 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87b0060220 con 0x7f87b00f5410 2024-02-19T22:19:17.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:17.392+0000 7f87ad7fa700 1 -- 172.21.15.88:0/78144752 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f87a4013380 con 0x7f87b00f5410 2024-02-19T22:19:17.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:17.392+0000 7f87ad7fa700 1 -- 172.21.15.88:0/78144752 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87a401ed80 con 0x7f87b00f5410 2024-02-19T22:19:17.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:17.392+0000 7f87b6eb9700 1 -- 172.21.15.88:0/78144752 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87b010bdf0 con 0x7f87b00f5410 2024-02-19T22:19:17.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:17.393+0000 7f87ad7fa700 1 -- 172.21.15.88:0/78144752 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f87a4014420 con 0x7f87b00f5410 2024-02-19T22:19:17.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:17.393+0000 7f87ad7fa700 1 --2- 172.21.15.88:0/78144752 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f87a002ff60 0x7f87a0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:17.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:17.393+0000 7f87b6eb9700 1 -- 172.21.15.88:0/78144752 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87b0045ad0 con 0x7f87b00f5410 2024-02-19T22:19:17.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:17.393+0000 7f87ad7fa700 1 -- 172.21.15.88:0/78144752 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f87a4040f00 con 0x7f87b00f5410 2024-02-19T22:19:17.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:17.394+0000 7f87affff700 1 --2- 172.21.15.88:0/78144752 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f87a002ff60 0x7f87a0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:17.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:17.394+0000 7f87affff700 1 --2- 172.21.15.88:0/78144752 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f87a002ff60 0x7f87a0032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f879c0096a0 tx=0x7f879c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:17.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:17.398+0000 7f87ad7fa700 1 -- 172.21.15.88:0/78144752 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f87a401c910 con 0x7f87b00f5410 2024-02-19T22:19:17.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:17 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:19:15.708827+0000 mgr.smithi052.ghqrtk (mgr.14162) 294 : cluster [DBG] pgmap v280: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:17.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:17.600+0000 7f87b6eb9700 1 -- 172.21.15.88:0/78144752 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f87b010c5e0 con 0x7f87b00f5410 2024-02-19T22:19:17.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:17.601+0000 7f87ad7fa700 1 -- 172.21.15.88:0/78144752 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f87a401c320 con 0x7f87b00f5410 2024-02-19T22:19:17.604 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:19:17.604 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:17.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:17.603+0000 7f87b6eb9700 1 -- 172.21.15.88:0/78144752 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f87a002ff60 msgr2=0x7f87a0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:17.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:17.603+0000 7f87b6eb9700 1 --2- 172.21.15.88:0/78144752 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f87a002ff60 0x7f87a0032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f879c0096a0 tx=0x7f879c006b40).stop 2024-02-19T22:19:17.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:17.603+0000 7f87b6eb9700 1 -- 172.21.15.88:0/78144752 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f87b00f5410 msgr2=0x7f87b010b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:17.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:17.603+0000 7f87b6eb9700 1 --2- 172.21.15.88:0/78144752 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f87b00f5410 0x7f87b010b900 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f87a40089f0 tx=0x7f87a400b040).stop 2024-02-19T22:19:17.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:17.603+0000 7f87b6eb9700 1 -- 172.21.15.88:0/78144752 shutdown_connections 2024-02-19T22:19:17.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:17.603+0000 7f87b6eb9700 1 --2- 172.21.15.88:0/78144752 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f87a002ff60 0x7f87a0032420 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:17.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:17.603+0000 7f87b6eb9700 1 --2- 172.21.15.88:0/78144752 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f87b00f5410 0x7f87b010b900 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:17.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:17.603+0000 7f87b6eb9700 1 -- 172.21.15.88:0/78144752 >> 172.21.15.88:0/78144752 conn(0x7f87b00f0b70 msgr2=0x7f87b00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:17.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:17.603+0000 7f87b6eb9700 1 -- 172.21.15.88:0/78144752 shutdown_connections 2024-02-19T22:19:17.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:17.603+0000 7f87b6eb9700 1 -- 172.21.15.88:0/78144752 wait complete. 2024-02-19T22:19:17.606 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:19:18.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:19:17.602521+0000 mon.smithi052 (mon.0) 288 : audit [DBG] from='client.? 172.21.15.88:0/78144752' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:19.262 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:19.262 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:19.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:19 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:19:17.709065+0000 mgr.smithi052.ghqrtk (mgr.14162) 295 : cluster [DBG] pgmap v281: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:21.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:21.054+0000 7f539810e700 1 -- 172.21.15.88:0/2866925989 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f53900f6510 msgr2=0x7f53900f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:21.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:21.054+0000 7f539810e700 1 --2- 172.21.15.88:0/2866925989 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f53900f6510 0x7f53900f6930 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f5380009fb0 tx=0x7f5380009b30).stop 2024-02-19T22:19:21.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:21.054+0000 7f539810e700 1 -- 172.21.15.88:0/2866925989 shutdown_connections 2024-02-19T22:19:21.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:21.054+0000 7f539810e700 1 --2- 172.21.15.88:0/2866925989 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f53900f6510 0x7f53900f6930 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:21.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:21.054+0000 7f539810e700 1 -- 172.21.15.88:0/2866925989 >> 172.21.15.88:0/2866925989 conn(0x7f53900f1c70 msgr2=0x7f53900f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:21.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:21.054+0000 7f539810e700 1 -- 172.21.15.88:0/2866925989 shutdown_connections 2024-02-19T22:19:21.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:21.054+0000 7f539810e700 1 -- 172.21.15.88:0/2866925989 wait complete. 2024-02-19T22:19:21.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:21.055+0000 7f539810e700 1 Processor -- start 2024-02-19T22:19:21.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:21.055+0000 7f539810e700 1 -- start start 2024-02-19T22:19:21.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:21.055+0000 7f539810e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f53900f6510 0x7f539006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:21.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:21.055+0000 7f539810e700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5390069e20 con 0x7f53900f6510 2024-02-19T22:19:21.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:21.055+0000 7f5395eaa700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f53900f6510 0x7f539006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:21.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:21.055+0000 7f5395eaa700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f53900f6510 0x7f539006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:49166/0 (socket says 172.21.15.88:49166) 2024-02-19T22:19:21.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:21.055+0000 7f5395eaa700 1 -- 172.21.15.88:0/1299738714 learned_addr learned my addr 172.21.15.88:0/1299738714 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:19:21.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:21.056+0000 7f5395eaa700 1 -- 172.21.15.88:0/1299738714 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5380005040 con 0x7f53900f6510 2024-02-19T22:19:21.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:21.056+0000 7f5395eaa700 1 --2- 172.21.15.88:0/1299738714 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f53900f6510 0x7f539006b9a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f5380005010 tx=0x7f538000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:21.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:21.056+0000 7f53867fc700 1 -- 172.21.15.88:0/1299738714 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5380009320 con 0x7f53900f6510 2024-02-19T22:19:21.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:21.056+0000 7f539810e700 1 -- 172.21.15.88:0/1299738714 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5390069fc0 con 0x7f53900f6510 2024-02-19T22:19:21.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:21.056+0000 7f53867fc700 1 -- 172.21.15.88:0/1299738714 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5380013380 con 0x7f53900f6510 2024-02-19T22:19:21.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:21.056+0000 7f53867fc700 1 -- 172.21.15.88:0/1299738714 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5380014e50 con 0x7f53900f6510 2024-02-19T22:19:21.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:21.056+0000 7f539810e700 1 -- 172.21.15.88:0/1299738714 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f539006a450 con 0x7f53900f6510 2024-02-19T22:19:21.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:21.057+0000 7f53867fc700 1 -- 172.21.15.88:0/1299738714 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f53800095f0 con 0x7f53900f6510 2024-02-19T22:19:21.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:21.057+0000 7f539810e700 1 -- 172.21.15.88:0/1299738714 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5390059100 con 0x7f53900f6510 2024-02-19T22:19:21.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:21.057+0000 7f53867fc700 1 --2- 172.21.15.88:0/1299738714 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f537c02ff10 0x7f537c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:21.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:21.057+0000 7f53867fc700 1 -- 172.21.15.88:0/1299738714 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5380040e70 con 0x7f53900f6510 2024-02-19T22:19:21.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:21.057+0000 7f53956a9700 1 --2- 172.21.15.88:0/1299738714 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f537c02ff10 0x7f537c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:21.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:21.057+0000 7f53956a9700 1 --2- 172.21.15.88:0/1299738714 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f537c02ff10 0x7f537c0323d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f538c0096a0 tx=0x7f538c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:21.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:21.062+0000 7f53867fc700 1 -- 172.21.15.88:0/1299738714 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5380014420 con 0x7f53900f6510 2024-02-19T22:19:21.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:21.264+0000 7f539810e700 1 -- 172.21.15.88:0/1299738714 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f53900fc0a0 con 0x7f53900f6510 2024-02-19T22:19:21.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:21.264+0000 7f53867fc700 1 -- 172.21.15.88:0/1299738714 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5380010230 con 0x7f53900f6510 2024-02-19T22:19:21.267 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:19:21.267 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:21.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:21.266+0000 7f539810e700 1 -- 172.21.15.88:0/1299738714 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f537c02ff10 msgr2=0x7f537c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:21.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:21.266+0000 7f539810e700 1 --2- 172.21.15.88:0/1299738714 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f537c02ff10 0x7f537c0323d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f538c0096a0 tx=0x7f538c006b40).stop 2024-02-19T22:19:21.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:21.266+0000 7f539810e700 1 -- 172.21.15.88:0/1299738714 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f53900f6510 msgr2=0x7f539006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:21.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:21.266+0000 7f539810e700 1 --2- 172.21.15.88:0/1299738714 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f53900f6510 0x7f539006b9a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f5380005010 tx=0x7f538000b040).stop 2024-02-19T22:19:21.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:21.266+0000 7f539810e700 1 -- 172.21.15.88:0/1299738714 shutdown_connections 2024-02-19T22:19:21.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:21.266+0000 7f539810e700 1 --2- 172.21.15.88:0/1299738714 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f537c02ff10 0x7f537c0323d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:21.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:21.266+0000 7f539810e700 1 --2- 172.21.15.88:0/1299738714 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f53900f6510 0x7f539006b9a0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:21.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:21.266+0000 7f539810e700 1 -- 172.21.15.88:0/1299738714 >> 172.21.15.88:0/1299738714 conn(0x7f53900f1c70 msgr2=0x7f53900f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:21.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:21.266+0000 7f539810e700 1 -- 172.21.15.88:0/1299738714 shutdown_connections 2024-02-19T22:19:21.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:21.266+0000 7f539810e700 1 -- 172.21.15.88:0/1299738714 wait complete. 2024-02-19T22:19:21.269 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:19:21.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:21 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:19:19.709342+0000 mgr.smithi052.ghqrtk (mgr.14162) 296 : cluster [DBG] pgmap v282: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:21.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:21 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:19:21.265790+0000 mon.smithi052 (mon.0) 289 : audit [DBG] from='client.? 172.21.15.88:0/1299738714' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:22.931 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:22.932 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:23.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:23 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:19:21.709652+0000 mgr.smithi052.ghqrtk (mgr.14162) 297 : cluster [DBG] pgmap v283: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:24.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:24.879+0000 7f39f6dcf700 1 -- 172.21.15.88:0/1160256415 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f39f00f6510 msgr2=0x7f39f00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:24.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:24.879+0000 7f39f6dcf700 1 --2- 172.21.15.88:0/1160256415 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f39f00f6510 0x7f39f00f6930 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f39e4009fb0 tx=0x7f39e4009b30).stop 2024-02-19T22:19:24.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:24.879+0000 7f39f6dcf700 1 -- 172.21.15.88:0/1160256415 shutdown_connections 2024-02-19T22:19:24.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:24.879+0000 7f39f6dcf700 1 --2- 172.21.15.88:0/1160256415 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f39f00f6510 0x7f39f00f6930 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:24.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:24.879+0000 7f39f6dcf700 1 -- 172.21.15.88:0/1160256415 >> 172.21.15.88:0/1160256415 conn(0x7f39f00f1c70 msgr2=0x7f39f00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:24.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:24.879+0000 7f39f6dcf700 1 -- 172.21.15.88:0/1160256415 shutdown_connections 2024-02-19T22:19:24.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:24.879+0000 7f39f6dcf700 1 -- 172.21.15.88:0/1160256415 wait complete. 2024-02-19T22:19:24.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:24.880+0000 7f39f6dcf700 1 Processor -- start 2024-02-19T22:19:24.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:24.880+0000 7f39f6dcf700 1 -- start start 2024-02-19T22:19:24.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:24.880+0000 7f39f6dcf700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f39f00f6510 0x7f39f006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:24.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:24.880+0000 7f39f6dcf700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f0069e20 con 0x7f39f00f6510 2024-02-19T22:19:24.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:24.880+0000 7f39f4b6b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f39f00f6510 0x7f39f006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:24.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:24.880+0000 7f39f4b6b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f39f00f6510 0x7f39f006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:56080/0 (socket says 172.21.15.88:56080) 2024-02-19T22:19:24.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:24.880+0000 7f39f4b6b700 1 -- 172.21.15.88:0/1040415640 learned_addr learned my addr 172.21.15.88:0/1040415640 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:19:24.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:24.881+0000 7f39f4b6b700 1 -- 172.21.15.88:0/1040415640 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e4005040 con 0x7f39f00f6510 2024-02-19T22:19:24.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:24.881+0000 7f39f4b6b700 1 --2- 172.21.15.88:0/1040415640 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f39f00f6510 0x7f39f006b9a0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f39e4005010 tx=0x7f39e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:24.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:24.881+0000 7f39ed7fa700 1 -- 172.21.15.88:0/1040415640 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39e4009320 con 0x7f39f00f6510 2024-02-19T22:19:24.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:24.881+0000 7f39f6dcf700 1 -- 172.21.15.88:0/1040415640 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39f0069fc0 con 0x7f39f00f6510 2024-02-19T22:19:24.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:24.881+0000 7f39ed7fa700 1 -- 172.21.15.88:0/1040415640 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f39e4013380 con 0x7f39f00f6510 2024-02-19T22:19:24.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:24.881+0000 7f39ed7fa700 1 -- 172.21.15.88:0/1040415640 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39e4014e50 con 0x7f39f00f6510 2024-02-19T22:19:24.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:24.881+0000 7f39f6dcf700 1 -- 172.21.15.88:0/1040415640 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39f006a450 con 0x7f39f00f6510 2024-02-19T22:19:24.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:24.882+0000 7f39ed7fa700 1 -- 172.21.15.88:0/1040415640 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f39e40095f0 con 0x7f39f00f6510 2024-02-19T22:19:24.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:24.882+0000 7f39f6dcf700 1 -- 172.21.15.88:0/1040415640 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39f0059100 con 0x7f39f00f6510 2024-02-19T22:19:24.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:24.882+0000 7f39ed7fa700 1 --2- 172.21.15.88:0/1040415640 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f39e002ff10 0x7f39e00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:24.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:24.882+0000 7f39ed7fa700 1 -- 172.21.15.88:0/1040415640 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f39e4040e90 con 0x7f39f00f6510 2024-02-19T22:19:24.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:24.882+0000 7f39effff700 1 --2- 172.21.15.88:0/1040415640 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f39e002ff10 0x7f39e00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:24.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:24.883+0000 7f39effff700 1 --2- 172.21.15.88:0/1040415640 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f39e002ff10 0x7f39e00323d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f39dc0096a0 tx=0x7f39dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:24.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:24.887+0000 7f39ed7fa700 1 -- 172.21.15.88:0/1040415640 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f39e4014420 con 0x7f39f00f6510 2024-02-19T22:19:25.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:25.094+0000 7f39f6dcf700 1 -- 172.21.15.88:0/1040415640 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f39f00fc0a0 con 0x7f39f00f6510 2024-02-19T22:19:25.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:25.095+0000 7f39ed7fa700 1 -- 172.21.15.88:0/1040415640 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f39e4010230 con 0x7f39f00f6510 2024-02-19T22:19:25.097 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:19:25.097 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:25.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:25.096+0000 7f39f6dcf700 1 -- 172.21.15.88:0/1040415640 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f39e002ff10 msgr2=0x7f39e00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:25.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:25.096+0000 7f39f6dcf700 1 --2- 172.21.15.88:0/1040415640 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f39e002ff10 0x7f39e00323d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f39dc0096a0 tx=0x7f39dc006b40).stop 2024-02-19T22:19:25.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:25.096+0000 7f39f6dcf700 1 -- 172.21.15.88:0/1040415640 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f39f00f6510 msgr2=0x7f39f006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:25.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:25.096+0000 7f39f6dcf700 1 --2- 172.21.15.88:0/1040415640 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f39f00f6510 0x7f39f006b9a0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f39e4005010 tx=0x7f39e400b040).stop 2024-02-19T22:19:25.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:25.097+0000 7f39f6dcf700 1 -- 172.21.15.88:0/1040415640 shutdown_connections 2024-02-19T22:19:25.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:25.097+0000 7f39f6dcf700 1 --2- 172.21.15.88:0/1040415640 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f39e002ff10 0x7f39e00323d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:25.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:25.097+0000 7f39f6dcf700 1 --2- 172.21.15.88:0/1040415640 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f39f00f6510 0x7f39f006b9a0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:25.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:25.097+0000 7f39f6dcf700 1 -- 172.21.15.88:0/1040415640 >> 172.21.15.88:0/1040415640 conn(0x7f39f00f1c70 msgr2=0x7f39f00f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:25.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:25.097+0000 7f39f6dcf700 1 -- 172.21.15.88:0/1040415640 shutdown_connections 2024-02-19T22:19:25.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:25.097+0000 7f39f6dcf700 1 -- 172.21.15.88:0/1040415640 wait complete. 2024-02-19T22:19:25.099 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:19:25.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:25 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:19:23.710013+0000 mgr.smithi052.ghqrtk (mgr.14162) 298 : cluster [DBG] pgmap v284: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:25.594 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:25 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:19:25.095364+0000 mon.smithi052 (mon.0) 290 : audit [DBG] from='client.? 172.21.15.88:0/1040415640' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:26.717 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:26.717 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:27.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:27 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:19:25.710412+0000 mgr.smithi052.ghqrtk (mgr.14162) 299 : cluster [DBG] pgmap v285: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:28.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:28.562+0000 7fde4bf1d700 1 -- 172.21.15.88:0/3995633402 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fde440f5410 msgr2=0x7fde440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:28.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:28.562+0000 7fde4bf1d700 1 --2- 172.21.15.88:0/3995633402 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fde440f5410 0x7fde440f5830 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7fde34009fb0 tx=0x7fde34009b30).stop 2024-02-19T22:19:28.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:28.562+0000 7fde4bf1d700 1 -- 172.21.15.88:0/3995633402 shutdown_connections 2024-02-19T22:19:28.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:28.562+0000 7fde4bf1d700 1 --2- 172.21.15.88:0/3995633402 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fde440f5410 0x7fde440f5830 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:28.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:28.562+0000 7fde4bf1d700 1 -- 172.21.15.88:0/3995633402 >> 172.21.15.88:0/3995633402 conn(0x7fde440f0b70 msgr2=0x7fde440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:28.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:28.562+0000 7fde4bf1d700 1 -- 172.21.15.88:0/3995633402 shutdown_connections 2024-02-19T22:19:28.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:28.562+0000 7fde4bf1d700 1 -- 172.21.15.88:0/3995633402 wait complete. 2024-02-19T22:19:28.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:28.563+0000 7fde4bf1d700 1 Processor -- start 2024-02-19T22:19:28.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:28.563+0000 7fde4bf1d700 1 -- start start 2024-02-19T22:19:28.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:28.563+0000 7fde4bf1d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fde440601e0 0x7fde4410ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:28.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:28.563+0000 7fde4bf1d700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde44107430 con 0x7fde440601e0 2024-02-19T22:19:28.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:28.563+0000 7fde49cb9700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fde440601e0 0x7fde4410ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:28.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:28.563+0000 7fde49cb9700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fde440601e0 0x7fde4410ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:56096/0 (socket says 172.21.15.88:56096) 2024-02-19T22:19:28.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:28.563+0000 7fde49cb9700 1 -- 172.21.15.88:0/2500225948 learned_addr learned my addr 172.21.15.88:0/2500225948 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:19:28.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:28.563+0000 7fde49cb9700 1 -- 172.21.15.88:0/2500225948 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde34005040 con 0x7fde440601e0 2024-02-19T22:19:28.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:28.564+0000 7fde49cb9700 1 --2- 172.21.15.88:0/2500225948 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fde440601e0 0x7fde4410ba40 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7fde34009f80 tx=0x7fde34009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:28.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:28.564+0000 7fde3a7fc700 1 -- 172.21.15.88:0/2500225948 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde34013380 con 0x7fde440601e0 2024-02-19T22:19:28.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:28.564+0000 7fde4bf1d700 1 -- 172.21.15.88:0/2500225948 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde4410bf30 con 0x7fde440601e0 2024-02-19T22:19:28.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:28.564+0000 7fde3a7fc700 1 -- 172.21.15.88:0/2500225948 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fde34013790 con 0x7fde440601e0 2024-02-19T22:19:28.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:28.564+0000 7fde3a7fc700 1 -- 172.21.15.88:0/2500225948 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde34014420 con 0x7fde440601e0 2024-02-19T22:19:28.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:28.564+0000 7fde4bf1d700 1 -- 172.21.15.88:0/2500225948 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde4410c280 con 0x7fde440601e0 2024-02-19T22:19:28.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:28.565+0000 7fde3a7fc700 1 -- 172.21.15.88:0/2500225948 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fde34025060 con 0x7fde440601e0 2024-02-19T22:19:28.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:28.565+0000 7fde4bf1d700 1 -- 172.21.15.88:0/2500225948 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde44045ad0 con 0x7fde440601e0 2024-02-19T22:19:28.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:28.565+0000 7fde3a7fc700 1 --2- 172.21.15.88:0/2500225948 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fde3002ff60 0x7fde30032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:28.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:28.565+0000 7fde3a7fc700 1 -- 172.21.15.88:0/2500225948 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fde34040d70 con 0x7fde440601e0 2024-02-19T22:19:28.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:28.565+0000 7fde494b8700 1 --2- 172.21.15.88:0/2500225948 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fde3002ff60 0x7fde30032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:28.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:28.565+0000 7fde494b8700 1 --2- 172.21.15.88:0/2500225948 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fde3002ff60 0x7fde30032420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fde400096a0 tx=0x7fde40006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:28.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:28.570+0000 7fde3a7fc700 1 -- 172.21.15.88:0/2500225948 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fde3400f280 con 0x7fde440601e0 2024-02-19T22:19:28.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:28.775+0000 7fde4bf1d700 1 -- 172.21.15.88:0/2500225948 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fde4410caf0 con 0x7fde440601e0 2024-02-19T22:19:28.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:28.776+0000 7fde3a7fc700 1 -- 172.21.15.88:0/2500225948 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fde34004490 con 0x7fde440601e0 2024-02-19T22:19:28.777 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:19:28.778 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:28.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:28.777+0000 7fde4bf1d700 1 -- 172.21.15.88:0/2500225948 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fde3002ff60 msgr2=0x7fde30032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:28.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:28.777+0000 7fde4bf1d700 1 --2- 172.21.15.88:0/2500225948 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fde3002ff60 0x7fde30032420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fde400096a0 tx=0x7fde40006b40).stop 2024-02-19T22:19:28.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:28.777+0000 7fde4bf1d700 1 -- 172.21.15.88:0/2500225948 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fde440601e0 msgr2=0x7fde4410ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:28.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:28.777+0000 7fde4bf1d700 1 --2- 172.21.15.88:0/2500225948 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fde440601e0 0x7fde4410ba40 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7fde34009f80 tx=0x7fde34009b30).stop 2024-02-19T22:19:28.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:28.777+0000 7fde4bf1d700 1 -- 172.21.15.88:0/2500225948 shutdown_connections 2024-02-19T22:19:28.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:28.777+0000 7fde4bf1d700 1 --2- 172.21.15.88:0/2500225948 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fde3002ff60 0x7fde30032420 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:28.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:28.777+0000 7fde4bf1d700 1 --2- 172.21.15.88:0/2500225948 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fde440601e0 0x7fde4410ba40 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:28.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:28.777+0000 7fde4bf1d700 1 -- 172.21.15.88:0/2500225948 >> 172.21.15.88:0/2500225948 conn(0x7fde440f0b70 msgr2=0x7fde440f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:28.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:28.778+0000 7fde4bf1d700 1 -- 172.21.15.88:0/2500225948 shutdown_connections 2024-02-19T22:19:28.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:28.778+0000 7fde4bf1d700 1 -- 172.21.15.88:0/2500225948 wait complete. 2024-02-19T22:19:28.780 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:19:29.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:29 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:19:27.710729+0000 mgr.smithi052.ghqrtk (mgr.14162) 300 : cluster [DBG] pgmap v286: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:29.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:29 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:19:28.776341+0000 mon.smithi052 (mon.0) 291 : audit [DBG] from='client.? 172.21.15.88:0/2500225948' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:30.529 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:30.529 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:31.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:31 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:19:29.711107+0000 mgr.smithi052.ghqrtk (mgr.14162) 301 : cluster [DBG] pgmap v287: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:32.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:32.472+0000 7f8dba7c8700 1 -- 172.21.15.88:0/3848030765 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8db4060220 msgr2=0x7f8db40f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:32.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:32.472+0000 7f8dba7c8700 1 --2- 172.21.15.88:0/3848030765 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8db4060220 0x7f8db40f9150 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f8d9c009fb0 tx=0x7f8d9c009b30).stop 2024-02-19T22:19:32.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:32.472+0000 7f8dba7c8700 1 -- 172.21.15.88:0/3848030765 shutdown_connections 2024-02-19T22:19:32.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:32.472+0000 7f8dba7c8700 1 --2- 172.21.15.88:0/3848030765 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8db4060220 0x7f8db40f9150 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:32.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:32.472+0000 7f8dba7c8700 1 -- 172.21.15.88:0/3848030765 >> 172.21.15.88:0/3848030765 conn(0x7f8db40f0b70 msgr2=0x7f8db40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:32.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:32.472+0000 7f8dba7c8700 1 -- 172.21.15.88:0/3848030765 shutdown_connections 2024-02-19T22:19:32.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:32.472+0000 7f8dba7c8700 1 -- 172.21.15.88:0/3848030765 wait complete. 2024-02-19T22:19:32.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:32.472+0000 7f8dba7c8700 1 Processor -- start 2024-02-19T22:19:32.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:32.472+0000 7f8dba7c8700 1 -- start start 2024-02-19T22:19:32.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:32.473+0000 7f8dba7c8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8db4060220 0x7f8db41067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:32.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:32.473+0000 7f8dba7c8700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db4106cb0 con 0x7f8db4060220 2024-02-19T22:19:32.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:32.473+0000 7f8db3fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8db4060220 0x7f8db41067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:32.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:32.473+0000 7f8db3fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8db4060220 0x7f8db41067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:56122/0 (socket says 172.21.15.88:56122) 2024-02-19T22:19:32.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:32.473+0000 7f8db3fff700 1 -- 172.21.15.88:0/894889699 learned_addr learned my addr 172.21.15.88:0/894889699 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:19:32.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:32.473+0000 7f8db3fff700 1 -- 172.21.15.88:0/894889699 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d9c005040 con 0x7f8db4060220 2024-02-19T22:19:32.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:32.473+0000 7f8db3fff700 1 --2- 172.21.15.88:0/894889699 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8db4060220 0x7f8db41067c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f8d9c0089f0 tx=0x7f8d9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:32.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:32.473+0000 7f8db0ff9700 1 -- 172.21.15.88:0/894889699 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d9c009320 con 0x7f8db4060220 2024-02-19T22:19:32.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:32.474+0000 7f8dba7c8700 1 -- 172.21.15.88:0/894889699 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8db4106eb0 con 0x7f8db4060220 2024-02-19T22:19:32.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:32.474+0000 7f8dba7c8700 1 -- 172.21.15.88:0/894889699 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8db4107340 con 0x7f8db4060220 2024-02-19T22:19:32.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:32.474+0000 7f8db0ff9700 1 -- 172.21.15.88:0/894889699 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8d9c013380 con 0x7f8db4060220 2024-02-19T22:19:32.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:32.474+0000 7f8db0ff9700 1 -- 172.21.15.88:0/894889699 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d9c014db0 con 0x7f8db4060220 2024-02-19T22:19:32.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:32.474+0000 7f8db0ff9700 1 -- 172.21.15.88:0/894889699 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8d9c014420 con 0x7f8db4060220 2024-02-19T22:19:32.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:32.474+0000 7f8dba7c8700 1 -- 172.21.15.88:0/894889699 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d98004fa0 con 0x7f8db4060220 2024-02-19T22:19:32.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:32.475+0000 7f8db0ff9700 1 --2- 172.21.15.88:0/894889699 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f8da002ff60 0x7f8da0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:32.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:32.475+0000 7f8db0ff9700 1 -- 172.21.15.88:0/894889699 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8d9c041040 con 0x7f8db4060220 2024-02-19T22:19:32.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:32.475+0000 7f8db37fe700 1 --2- 172.21.15.88:0/894889699 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f8da002ff60 0x7f8da0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:32.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:32.475+0000 7f8db37fe700 1 --2- 172.21.15.88:0/894889699 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f8da002ff60 0x7f8da0032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f8da40096a0 tx=0x7f8da4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:32.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:32.480+0000 7f8db0ff9700 1 -- 172.21.15.88:0/894889699 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8d9c012d00 con 0x7f8db4060220 2024-02-19T22:19:32.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:32.683+0000 7f8dba7c8700 1 -- 172.21.15.88:0/894889699 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8d98005670 con 0x7f8db4060220 2024-02-19T22:19:32.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:32.684+0000 7f8db0ff9700 1 -- 172.21.15.88:0/894889699 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8d9c012b20 con 0x7f8db4060220 2024-02-19T22:19:32.685 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:19:32.685 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:32.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:32.685+0000 7f8dba7c8700 1 -- 172.21.15.88:0/894889699 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f8da002ff60 msgr2=0x7f8da0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:32.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:32.685+0000 7f8dba7c8700 1 --2- 172.21.15.88:0/894889699 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f8da002ff60 0x7f8da0032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f8da40096a0 tx=0x7f8da4006b40).stop 2024-02-19T22:19:32.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:32.685+0000 7f8dba7c8700 1 -- 172.21.15.88:0/894889699 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8db4060220 msgr2=0x7f8db41067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:32.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:32.685+0000 7f8dba7c8700 1 --2- 172.21.15.88:0/894889699 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8db4060220 0x7f8db41067c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f8d9c0089f0 tx=0x7f8d9c00b040).stop 2024-02-19T22:19:32.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:32.685+0000 7f8dba7c8700 1 -- 172.21.15.88:0/894889699 shutdown_connections 2024-02-19T22:19:32.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:32.685+0000 7f8dba7c8700 1 --2- 172.21.15.88:0/894889699 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f8da002ff60 0x7f8da0032420 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:32.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:32.685+0000 7f8dba7c8700 1 --2- 172.21.15.88:0/894889699 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8db4060220 0x7f8db41067c0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:32.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:32.685+0000 7f8dba7c8700 1 -- 172.21.15.88:0/894889699 >> 172.21.15.88:0/894889699 conn(0x7f8db40f0b70 msgr2=0x7f8db40f1620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:32.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:32.685+0000 7f8dba7c8700 1 -- 172.21.15.88:0/894889699 shutdown_connections 2024-02-19T22:19:32.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:32.685+0000 7f8dba7c8700 1 -- 172.21.15.88:0/894889699 wait complete. 2024-02-19T22:19:32.688 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:19:33.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:33 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:19:31.711430+0000 mgr.smithi052.ghqrtk (mgr.14162) 302 : cluster [DBG] pgmap v288: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:33.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:33 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:19:32.684093+0000 mon.smithi052 (mon.0) 292 : audit [DBG] from='client.? 172.21.15.88:0/894889699' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:34.437 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:34.438 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:35.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:35 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:19:33.711776+0000 mgr.smithi052.ghqrtk (mgr.14162) 303 : cluster [DBG] pgmap v289: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:36.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:36.184+0000 7f94e9976700 1 -- 172.21.15.88:0/360707568 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94e40f6510 msgr2=0x7f94e40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:36.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:36.184+0000 7f94e9976700 1 --2- 172.21.15.88:0/360707568 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94e40f6510 0x7f94e40f6930 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f94d8009fb0 tx=0x7f94d8009b30).stop 2024-02-19T22:19:36.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:36.184+0000 7f94e9976700 1 -- 172.21.15.88:0/360707568 shutdown_connections 2024-02-19T22:19:36.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:36.184+0000 7f94e9976700 1 --2- 172.21.15.88:0/360707568 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94e40f6510 0x7f94e40f6930 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:36.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:36.184+0000 7f94e9976700 1 -- 172.21.15.88:0/360707568 >> 172.21.15.88:0/360707568 conn(0x7f94e40f1c70 msgr2=0x7f94e40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:36.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:36.184+0000 7f94e9976700 1 -- 172.21.15.88:0/360707568 shutdown_connections 2024-02-19T22:19:36.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:36.184+0000 7f94e9976700 1 -- 172.21.15.88:0/360707568 wait complete. 2024-02-19T22:19:36.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:36.185+0000 7f94e9976700 1 Processor -- start 2024-02-19T22:19:36.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:36.185+0000 7f94e9976700 1 -- start start 2024-02-19T22:19:36.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:36.185+0000 7f94e9976700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94e40f6510 0x7f94e41067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:36.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:36.185+0000 7f94e9976700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94e4106cd0 con 0x7f94e40f6510 2024-02-19T22:19:36.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:36.185+0000 7f94e2ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94e40f6510 0x7f94e41067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:36.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:36.185+0000 7f94e2ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94e40f6510 0x7f94e41067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:40508/0 (socket says 172.21.15.88:40508) 2024-02-19T22:19:36.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:36.185+0000 7f94e2ffd700 1 -- 172.21.15.88:0/818667014 learned_addr learned my addr 172.21.15.88:0/818667014 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:19:36.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:36.185+0000 7f94e2ffd700 1 -- 172.21.15.88:0/818667014 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94d8005040 con 0x7f94e40f6510 2024-02-19T22:19:36.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:36.186+0000 7f94e2ffd700 1 --2- 172.21.15.88:0/818667014 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94e40f6510 0x7f94e41067e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f94d8005010 tx=0x7f94d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:36.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:36.186+0000 7f94cb7fe700 1 -- 172.21.15.88:0/818667014 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94d8009320 con 0x7f94e40f6510 2024-02-19T22:19:36.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:36.186+0000 7f94e9976700 1 -- 172.21.15.88:0/818667014 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94e4106ed0 con 0x7f94e40f6510 2024-02-19T22:19:36.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:36.186+0000 7f94e9976700 1 -- 172.21.15.88:0/818667014 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94e4107360 con 0x7f94e40f6510 2024-02-19T22:19:36.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:36.186+0000 7f94cb7fe700 1 -- 172.21.15.88:0/818667014 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f94d8013380 con 0x7f94e40f6510 2024-02-19T22:19:36.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:36.186+0000 7f94cb7fe700 1 -- 172.21.15.88:0/818667014 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94d8014df0 con 0x7f94e40f6510 2024-02-19T22:19:36.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:36.187+0000 7f94e9976700 1 -- 172.21.15.88:0/818667014 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94e4045ad0 con 0x7f94e40f6510 2024-02-19T22:19:36.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:36.187+0000 7f94cb7fe700 1 -- 172.21.15.88:0/818667014 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f94d8014420 con 0x7f94e40f6510 2024-02-19T22:19:36.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:36.187+0000 7f94cb7fe700 1 --2- 172.21.15.88:0/818667014 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f94cc02ff10 0x7f94cc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:36.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:36.187+0000 7f94cb7fe700 1 -- 172.21.15.88:0/818667014 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f94d80410e0 con 0x7f94e40f6510 2024-02-19T22:19:36.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:36.187+0000 7f94e27fc700 1 --2- 172.21.15.88:0/818667014 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f94cc02ff10 0x7f94cc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:36.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:36.187+0000 7f94e27fc700 1 --2- 172.21.15.88:0/818667014 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f94cc02ff10 0x7f94cc0323d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f94d40096a0 tx=0x7f94d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:36.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:36.192+0000 7f94cb7fe700 1 -- 172.21.15.88:0/818667014 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f94d800fd10 con 0x7f94e40f6510 2024-02-19T22:19:36.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:36.398+0000 7f94e9976700 1 -- 172.21.15.88:0/818667014 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f94e4108210 con 0x7f94e40f6510 2024-02-19T22:19:36.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:36.399+0000 7f94cb7fe700 1 -- 172.21.15.88:0/818667014 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f94d800fb30 con 0x7f94e40f6510 2024-02-19T22:19:36.401 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:19:36.401 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:36.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:36.400+0000 7f94e9976700 1 -- 172.21.15.88:0/818667014 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f94cc02ff10 msgr2=0x7f94cc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:36.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:36.400+0000 7f94e9976700 1 --2- 172.21.15.88:0/818667014 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f94cc02ff10 0x7f94cc0323d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f94d40096a0 tx=0x7f94d4006b40).stop 2024-02-19T22:19:36.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:36.400+0000 7f94e9976700 1 -- 172.21.15.88:0/818667014 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94e40f6510 msgr2=0x7f94e41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:36.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:36.400+0000 7f94e9976700 1 --2- 172.21.15.88:0/818667014 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94e40f6510 0x7f94e41067e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f94d8005010 tx=0x7f94d800b040).stop 2024-02-19T22:19:36.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:36.400+0000 7f94e9976700 1 -- 172.21.15.88:0/818667014 shutdown_connections 2024-02-19T22:19:36.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:36.400+0000 7f94e9976700 1 --2- 172.21.15.88:0/818667014 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f94cc02ff10 0x7f94cc0323d0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:36.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:36.400+0000 7f94e9976700 1 --2- 172.21.15.88:0/818667014 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94e40f6510 0x7f94e41067e0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:36.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:36.400+0000 7f94e9976700 1 -- 172.21.15.88:0/818667014 >> 172.21.15.88:0/818667014 conn(0x7f94e40f1c70 msgr2=0x7f94e40f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:36.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:36.400+0000 7f94e9976700 1 -- 172.21.15.88:0/818667014 shutdown_connections 2024-02-19T22:19:36.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:36.400+0000 7f94e9976700 1 -- 172.21.15.88:0/818667014 wait complete. 2024-02-19T22:19:36.404 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:19:37.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:37 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:19:35.712131+0000 mgr.smithi052.ghqrtk (mgr.14162) 304 : cluster [DBG] pgmap v290: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:37.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:37 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:19:36.400124+0000 mon.smithi052 (mon.0) 293 : audit [DBG] from='client.? 172.21.15.88:0/818667014' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:38.247 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:38.247 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:39.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:39 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:19:37.712379+0000 mgr.smithi052.ghqrtk (mgr.14162) 305 : cluster [DBG] pgmap v291: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:40.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:40.098+0000 7fb431a63700 1 -- 172.21.15.88:0/1004053097 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb42c0f5410 msgr2=0x7fb42c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:40.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:40.098+0000 7fb431a63700 1 --2- 172.21.15.88:0/1004053097 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb42c0f5410 0x7fb42c0f5830 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7fb420009fb0 tx=0x7fb420009b30).stop 2024-02-19T22:19:40.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:40.098+0000 7fb431a63700 1 -- 172.21.15.88:0/1004053097 shutdown_connections 2024-02-19T22:19:40.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:40.098+0000 7fb431a63700 1 --2- 172.21.15.88:0/1004053097 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb42c0f5410 0x7fb42c0f5830 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:40.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:40.098+0000 7fb431a63700 1 -- 172.21.15.88:0/1004053097 >> 172.21.15.88:0/1004053097 conn(0x7fb42c0f0b70 msgr2=0x7fb42c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:40.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:40.098+0000 7fb431a63700 1 -- 172.21.15.88:0/1004053097 shutdown_connections 2024-02-19T22:19:40.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:40.098+0000 7fb431a63700 1 -- 172.21.15.88:0/1004053097 wait complete. 2024-02-19T22:19:40.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:40.099+0000 7fb431a63700 1 Processor -- start 2024-02-19T22:19:40.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:40.099+0000 7fb431a63700 1 -- start start 2024-02-19T22:19:40.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:40.099+0000 7fb431a63700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb42c0f5410 0x7fb42c1089f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:40.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:40.099+0000 7fb431a63700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42c108ee0 con 0x7fb42c0f5410 2024-02-19T22:19:40.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:40.099+0000 7fb42b7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb42c0f5410 0x7fb42c1089f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:40.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:40.099+0000 7fb42b7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb42c0f5410 0x7fb42c1089f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:40530/0 (socket says 172.21.15.88:40530) 2024-02-19T22:19:40.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:40.099+0000 7fb42b7fe700 1 -- 172.21.15.88:0/127040947 learned_addr learned my addr 172.21.15.88:0/127040947 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:19:40.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:40.100+0000 7fb42b7fe700 1 -- 172.21.15.88:0/127040947 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb420005040 con 0x7fb42c0f5410 2024-02-19T22:19:40.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:40.100+0000 7fb42b7fe700 1 --2- 172.21.15.88:0/127040947 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb42c0f5410 0x7fb42c1089f0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7fb4200089f0 tx=0x7fb42000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:40.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:40.100+0000 7fb413fff700 1 -- 172.21.15.88:0/127040947 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb420009320 con 0x7fb42c0f5410 2024-02-19T22:19:40.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:40.100+0000 7fb431a63700 1 -- 172.21.15.88:0/127040947 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb42c1090e0 con 0x7fb42c0f5410 2024-02-19T22:19:40.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:40.100+0000 7fb413fff700 1 -- 172.21.15.88:0/127040947 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb420013380 con 0x7fb42c0f5410 2024-02-19T22:19:40.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:40.100+0000 7fb413fff700 1 -- 172.21.15.88:0/127040947 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb42001ed80 con 0x7fb42c0f5410 2024-02-19T22:19:40.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:40.100+0000 7fb431a63700 1 -- 172.21.15.88:0/127040947 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb42c109570 con 0x7fb42c0f5410 2024-02-19T22:19:40.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:40.101+0000 7fb413fff700 1 -- 172.21.15.88:0/127040947 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb42001e420 con 0x7fb42c0f5410 2024-02-19T22:19:40.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:40.101+0000 7fb431a63700 1 -- 172.21.15.88:0/127040947 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb42c045ad0 con 0x7fb42c0f5410 2024-02-19T22:19:40.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:40.101+0000 7fb413fff700 1 --2- 172.21.15.88:0/127040947 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fb41402ffb0 0x7fb414032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:40.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:40.101+0000 7fb413fff700 1 -- 172.21.15.88:0/127040947 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb420040fc0 con 0x7fb42c0f5410 2024-02-19T22:19:40.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:40.101+0000 7fb42affd700 1 --2- 172.21.15.88:0/127040947 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fb41402ffb0 0x7fb414032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:40.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:40.102+0000 7fb42affd700 1 --2- 172.21.15.88:0/127040947 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fb41402ffb0 0x7fb414032470 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fb41c0096a0 tx=0x7fb41c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:40.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:40.106+0000 7fb413fff700 1 -- 172.21.15.88:0/127040947 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb42006d2b0 con 0x7fb42c0f5410 2024-02-19T22:19:40.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:40.311+0000 7fb431a63700 1 -- 172.21.15.88:0/127040947 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb42c0fafa0 con 0x7fb42c0f5410 2024-02-19T22:19:40.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:40.311+0000 7fb413fff700 1 -- 172.21.15.88:0/127040947 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb4200095f0 con 0x7fb42c0f5410 2024-02-19T22:19:40.314 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:19:40.314 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:40.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:40.313+0000 7fb431a63700 1 -- 172.21.15.88:0/127040947 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fb41402ffb0 msgr2=0x7fb414032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:40.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:40.313+0000 7fb431a63700 1 --2- 172.21.15.88:0/127040947 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fb41402ffb0 0x7fb414032470 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fb41c0096a0 tx=0x7fb41c006b40).stop 2024-02-19T22:19:40.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:40.313+0000 7fb431a63700 1 -- 172.21.15.88:0/127040947 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb42c0f5410 msgr2=0x7fb42c1089f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:40.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:40.313+0000 7fb431a63700 1 --2- 172.21.15.88:0/127040947 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb42c0f5410 0x7fb42c1089f0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7fb4200089f0 tx=0x7fb42000b040).stop 2024-02-19T22:19:40.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:40.313+0000 7fb431a63700 1 -- 172.21.15.88:0/127040947 shutdown_connections 2024-02-19T22:19:40.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:40.313+0000 7fb431a63700 1 --2- 172.21.15.88:0/127040947 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fb41402ffb0 0x7fb414032470 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:40.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:40.313+0000 7fb431a63700 1 --2- 172.21.15.88:0/127040947 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb42c0f5410 0x7fb42c1089f0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:40.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:40.313+0000 7fb431a63700 1 -- 172.21.15.88:0/127040947 >> 172.21.15.88:0/127040947 conn(0x7fb42c0f0b70 msgr2=0x7fb42c05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:40.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:40.313+0000 7fb431a63700 1 -- 172.21.15.88:0/127040947 shutdown_connections 2024-02-19T22:19:40.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:40.313+0000 7fb431a63700 1 -- 172.21.15.88:0/127040947 wait complete. 2024-02-19T22:19:40.316 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:19:41.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:41 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:19:39.712736+0000 mgr.smithi052.ghqrtk (mgr.14162) 306 : cluster [DBG] pgmap v292: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:41.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:41 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:19:40.312783+0000 mon.smithi052 (mon.0) 294 : audit [DBG] from='client.? 172.21.15.88:0/127040947' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:41.971 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:41.971 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:43.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:43 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:19:41.713025+0000 mgr.smithi052.ghqrtk (mgr.14162) 307 : cluster [DBG] pgmap v293: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:43.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:43.800+0000 7fc5d3448700 1 -- 172.21.15.88:0/3355364711 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc5cc060280 msgr2=0x7fc5cc0f6ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:43.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:43.800+0000 7fc5d3448700 1 --2- 172.21.15.88:0/3355364711 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc5cc060280 0x7fc5cc0f6ee0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7fc5c8009fb0 tx=0x7fc5c8009b30).stop 2024-02-19T22:19:43.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:43.800+0000 7fc5d3448700 1 -- 172.21.15.88:0/3355364711 shutdown_connections 2024-02-19T22:19:43.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:43.800+0000 7fc5d3448700 1 --2- 172.21.15.88:0/3355364711 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc5cc060280 0x7fc5cc0f6ee0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:43.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:43.800+0000 7fc5d3448700 1 -- 172.21.15.88:0/3355364711 >> 172.21.15.88:0/3355364711 conn(0x7fc5cc0f0b70 msgr2=0x7fc5cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:43.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:43.800+0000 7fc5d3448700 1 -- 172.21.15.88:0/3355364711 shutdown_connections 2024-02-19T22:19:43.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:43.800+0000 7fc5d3448700 1 -- 172.21.15.88:0/3355364711 wait complete. 2024-02-19T22:19:43.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:43.801+0000 7fc5d3448700 1 Processor -- start 2024-02-19T22:19:43.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:43.801+0000 7fc5d3448700 1 -- start start 2024-02-19T22:19:43.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:43.801+0000 7fc5d3448700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc5cc060280 0x7fc5cc10cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:43.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:43.801+0000 7fc5d3448700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5cc10d2c0 con 0x7fc5cc060280 2024-02-19T22:19:43.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:43.801+0000 7fc5d11e4700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc5cc060280 0x7fc5cc10cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:43.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:43.801+0000 7fc5d11e4700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc5cc060280 0x7fc5cc10cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:52858/0 (socket says 172.21.15.88:52858) 2024-02-19T22:19:43.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:43.801+0000 7fc5d11e4700 1 -- 172.21.15.88:0/483916139 learned_addr learned my addr 172.21.15.88:0/483916139 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:19:43.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:43.802+0000 7fc5d11e4700 1 -- 172.21.15.88:0/483916139 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c8005040 con 0x7fc5cc060280 2024-02-19T22:19:43.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:43.802+0000 7fc5d11e4700 1 --2- 172.21.15.88:0/483916139 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc5cc060280 0x7fc5cc10cdd0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7fc5c80089f0 tx=0x7fc5c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:43.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:43.802+0000 7fc5c1ffb700 1 -- 172.21.15.88:0/483916139 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5c8009320 con 0x7fc5cc060280 2024-02-19T22:19:43.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:43.802+0000 7fc5d3448700 1 -- 172.21.15.88:0/483916139 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5cc10d4c0 con 0x7fc5cc060280 2024-02-19T22:19:43.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:43.802+0000 7fc5c1ffb700 1 -- 172.21.15.88:0/483916139 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc5c8013380 con 0x7fc5cc060280 2024-02-19T22:19:43.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:43.802+0000 7fc5c1ffb700 1 -- 172.21.15.88:0/483916139 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5c8014e20 con 0x7fc5cc060280 2024-02-19T22:19:43.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:43.802+0000 7fc5d3448700 1 -- 172.21.15.88:0/483916139 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5cc10e100 con 0x7fc5cc060280 2024-02-19T22:19:43.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:43.803+0000 7fc5d3448700 1 -- 172.21.15.88:0/483916139 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5cc10d950 con 0x7fc5cc060280 2024-02-19T22:19:43.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:43.803+0000 7fc5c1ffb700 1 -- 172.21.15.88:0/483916139 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc5c80095f0 con 0x7fc5cc060280 2024-02-19T22:19:43.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:43.803+0000 7fc5c1ffb700 1 --2- 172.21.15.88:0/483916139 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fc5b802ff10 0x7fc5b80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:43.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:43.803+0000 7fc5c1ffb700 1 -- 172.21.15.88:0/483916139 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc5c8040f40 con 0x7fc5cc060280 2024-02-19T22:19:43.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:43.803+0000 7fc5d09e3700 1 --2- 172.21.15.88:0/483916139 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fc5b802ff10 0x7fc5b80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:43.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:43.804+0000 7fc5d09e3700 1 --2- 172.21.15.88:0/483916139 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fc5b802ff10 0x7fc5b80323d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fc5bc0096a0 tx=0x7fc5bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:43.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:43.808+0000 7fc5c1ffb700 1 -- 172.21.15.88:0/483916139 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc5c8014420 con 0x7fc5cc060280 2024-02-19T22:19:44.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:44.013+0000 7fc5d3448700 1 -- 172.21.15.88:0/483916139 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc5cc10e820 con 0x7fc5cc060280 2024-02-19T22:19:44.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:44.014+0000 7fc5c1ffb700 1 -- 172.21.15.88:0/483916139 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc5c8010230 con 0x7fc5cc060280 2024-02-19T22:19:44.016 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:19:44.016 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:44.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:44.015+0000 7fc5d3448700 1 -- 172.21.15.88:0/483916139 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fc5b802ff10 msgr2=0x7fc5b80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:44.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:44.015+0000 7fc5d3448700 1 --2- 172.21.15.88:0/483916139 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fc5b802ff10 0x7fc5b80323d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fc5bc0096a0 tx=0x7fc5bc006b40).stop 2024-02-19T22:19:44.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:44.015+0000 7fc5d3448700 1 -- 172.21.15.88:0/483916139 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc5cc060280 msgr2=0x7fc5cc10cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:44.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:44.015+0000 7fc5d3448700 1 --2- 172.21.15.88:0/483916139 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc5cc060280 0x7fc5cc10cdd0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7fc5c80089f0 tx=0x7fc5c800b040).stop 2024-02-19T22:19:44.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:44.015+0000 7fc5d3448700 1 -- 172.21.15.88:0/483916139 shutdown_connections 2024-02-19T22:19:44.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:44.015+0000 7fc5d3448700 1 --2- 172.21.15.88:0/483916139 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fc5b802ff10 0x7fc5b80323d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:44.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:44.015+0000 7fc5d3448700 1 --2- 172.21.15.88:0/483916139 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc5cc060280 0x7fc5cc10cdd0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:44.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:44.015+0000 7fc5d3448700 1 -- 172.21.15.88:0/483916139 >> 172.21.15.88:0/483916139 conn(0x7fc5cc0f0b70 msgr2=0x7fc5cc0f1770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:44.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:44.015+0000 7fc5d3448700 1 -- 172.21.15.88:0/483916139 shutdown_connections 2024-02-19T22:19:44.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:44.015+0000 7fc5d3448700 1 -- 172.21.15.88:0/483916139 wait complete. 2024-02-19T22:19:44.019 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:19:44.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:44 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:19:43.755180+0000 mon.smithi052 (mon.0) 295 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi052.ghqrtk/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:19:44.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:44 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:19:43.757523+0000 mon.smithi052 (mon.0) 296 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi052.ghqrtk/trash_purge_schedule"}]: dispatch 2024-02-19T22:19:44.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:44 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:19:44.015020+0000 mon.smithi052 (mon.0) 297 : audit [DBG] from='client.? 172.21.15.88:0/483916139' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:45.594 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:45 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:19:43.713398+0000 mgr.smithi052.ghqrtk (mgr.14162) 308 : cluster [DBG] pgmap v294: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:45.594 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:45 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:19:44.506177+0000 mon.smithi052 (mon.0) 298 : audit [DBG] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T22:19:45.594 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:45 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:19:44.741575+0000 mon.smithi052 (mon.0) 299 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' 2024-02-19T22:19:45.594 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:45 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:19:44.743121+0000 mon.smithi052 (mon.0) 300 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' 2024-02-19T22:19:45.594 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:45 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:19:44.971760+0000 mon.smithi052 (mon.0) 301 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' 2024-02-19T22:19:45.633 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:45.634 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:46.594 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cephadm 2024-02-19T22:19:44.505577+0000 mgr.smithi052.ghqrtk (mgr.14162) 309 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi052 ... 2024-02-19T22:19:46.594 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi052 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-02-19T22:19:46.594 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-02-19T22:19:46.594 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: stat: stderr time="2024-02-19T22:17:51Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:19:46.594 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: stat: stderr time="2024-02-19T22:18:29Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:19:46.594 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: stat: stderr time="2024-02-19T22:19:07Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:19:46.594 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-19T22:19:46.594 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: Traceback (most recent call last): 2024-02-19T22:19:46.594 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: File "/var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-19T22:19:46.595 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: main() 2024-02-19T22:19:46.595 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: File "/var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-19T22:19:46.595 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: r = ctx.func(ctx) 2024-02-19T22:19:46.595 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: File "/var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-19T22:19:46.595 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: return func(ctx) 2024-02-19T22:19:46.595 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: File "/var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-19T22:19:46.595 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-19T22:19:46.595 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: File "/var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-02-19T22:19:46.595 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-02-19T22:19:46.595 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: File "/var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-19T22:19:46.595 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: raise RuntimeError('uid/gid not found') 2024-02-19T22:19:46.595 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: RuntimeError: uid/gid not found 2024-02-19T22:19:46.595 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: Traceback (most recent call last): 2024-02-19T22:19:46.595 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-02-19T22:19:46.595 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: yield (conn, connr) 2024-02-19T22:19:46.595 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-02-19T22:19:46.596 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: code, '\n'.join(err))) 2024-02-19T22:19:46.596 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi052 ... 2024-02-19T22:19:46.596 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi052 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-02-19T22:19:46.596 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-02-19T22:19:46.596 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: stat: stderr time="2024-02-19T22:17:51Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:19:46.596 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: stat: stderr time="2024-02-19T22:18:29Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:19:46.596 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: stat: stderr time="2024-02-19T22:19:07Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:19:46.596 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-19T22:19:46.596 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: Traceback (most recent call last): 2024-02-19T22:19:46.596 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: File "/var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-19T22:19:46.596 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: main() 2024-02-19T22:19:46.596 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: File "/var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-19T22:19:46.596 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: r = ctx.func(ctx) 2024-02-19T22:19:46.596 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: File "/var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-19T22:19:46.596 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: return func(ctx) 2024-02-19T22:19:46.597 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: File "/var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-19T22:19:46.597 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-19T22:19:46.597 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: File "/var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-02-19T22:19:46.597 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-02-19T22:19:46.597 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: File "/var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-19T22:19:46.597 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: raise RuntimeError('uid/gid not found') 2024-02-19T22:19:46.597 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: RuntimeError: uid/gid not found 2024-02-19T22:19:47.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:47 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:19:45.713741+0000 mgr.smithi052.ghqrtk (mgr.14162) 310 : cluster [DBG] pgmap v295: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:47.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:47 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:19:46.795427+0000 mon.smithi052 (mon.0) 302 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' 2024-02-19T22:19:48.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:48.197+0000 7f9dd5be4700 1 -- 172.21.15.88:0/2330918150 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9dd0060200 msgr2=0x7f9dd00f9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:48.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:48.197+0000 7f9dd5be4700 1 --2- 172.21.15.88:0/2330918150 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9dd0060200 0x7f9dd00f9170 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f9dc0009fb0 tx=0x7f9dc0009b30).stop 2024-02-19T22:19:48.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:48.197+0000 7f9dd5be4700 1 -- 172.21.15.88:0/2330918150 shutdown_connections 2024-02-19T22:19:48.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:48.197+0000 7f9dd5be4700 1 --2- 172.21.15.88:0/2330918150 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9dd0060200 0x7f9dd00f9170 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:48.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:48.197+0000 7f9dd5be4700 1 -- 172.21.15.88:0/2330918150 >> 172.21.15.88:0/2330918150 conn(0x7f9dd00f0b70 msgr2=0x7f9dd00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:48.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:48.197+0000 7f9dd5be4700 1 -- 172.21.15.88:0/2330918150 shutdown_connections 2024-02-19T22:19:48.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:48.197+0000 7f9dd5be4700 1 -- 172.21.15.88:0/2330918150 wait complete. 2024-02-19T22:19:48.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:48.198+0000 7f9dd5be4700 1 Processor -- start 2024-02-19T22:19:48.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:48.198+0000 7f9dd5be4700 1 -- start start 2024-02-19T22:19:48.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:48.198+0000 7f9dd5be4700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9dd0060200 0x7f9dd0106770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:48.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:48.198+0000 7f9dd5be4700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd0106c60 con 0x7f9dd0060200 2024-02-19T22:19:48.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:48.198+0000 7f9dcf7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9dd0060200 0x7f9dd0106770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:48.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:48.198+0000 7f9dcf7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9dd0060200 0x7f9dd0106770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:52862/0 (socket says 172.21.15.88:52862) 2024-02-19T22:19:48.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:48.199+0000 7f9dcf7fe700 1 -- 172.21.15.88:0/3021339757 learned_addr learned my addr 172.21.15.88:0/3021339757 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:19:48.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:48.199+0000 7f9dcf7fe700 1 -- 172.21.15.88:0/3021339757 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dc0005040 con 0x7f9dd0060200 2024-02-19T22:19:48.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:48.199+0000 7f9dcf7fe700 1 --2- 172.21.15.88:0/3021339757 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9dd0060200 0x7f9dd0106770 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f9dc00089f0 tx=0x7f9dc000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:48.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:48.199+0000 7f9db7fff700 1 -- 172.21.15.88:0/3021339757 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9dc0009320 con 0x7f9dd0060200 2024-02-19T22:19:48.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:48.199+0000 7f9dd5be4700 1 -- 172.21.15.88:0/3021339757 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dd0106e60 con 0x7f9dd0060200 2024-02-19T22:19:48.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:48.199+0000 7f9db7fff700 1 -- 172.21.15.88:0/3021339757 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9dc0013380 con 0x7f9dd0060200 2024-02-19T22:19:48.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:48.199+0000 7f9db7fff700 1 -- 172.21.15.88:0/3021339757 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9dc0014e20 con 0x7f9dd0060200 2024-02-19T22:19:48.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:48.199+0000 7f9dd5be4700 1 -- 172.21.15.88:0/3021339757 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dd01072f0 con 0x7f9dd0060200 2024-02-19T22:19:48.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:48.200+0000 7f9db7fff700 1 -- 172.21.15.88:0/3021339757 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9dc00263a0 con 0x7f9dd0060200 2024-02-19T22:19:48.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:48.200+0000 7f9dd5be4700 1 -- 172.21.15.88:0/3021339757 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dd0045ad0 con 0x7f9dd0060200 2024-02-19T22:19:48.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:48.200+0000 7f9db7fff700 1 --2- 172.21.15.88:0/3021339757 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f9db802ff10 0x7f9db80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:48.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:48.200+0000 7f9db7fff700 1 -- 172.21.15.88:0/3021339757 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9dc0040f60 con 0x7f9dd0060200 2024-02-19T22:19:48.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:48.200+0000 7f9dceffd700 1 --2- 172.21.15.88:0/3021339757 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f9db802ff10 0x7f9db80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:48.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:48.201+0000 7f9dceffd700 1 --2- 172.21.15.88:0/3021339757 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f9db802ff10 0x7f9db80323d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f9dc40096a0 tx=0x7f9dc4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:48.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:48.205+0000 7f9db7fff700 1 -- 172.21.15.88:0/3021339757 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9dc00126f0 con 0x7f9dd0060200 2024-02-19T22:19:48.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:48.420+0000 7f9dd5be4700 1 -- 172.21.15.88:0/3021339757 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9dd0108140 con 0x7f9dd0060200 2024-02-19T22:19:48.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:48.421+0000 7f9db7fff700 1 -- 172.21.15.88:0/3021339757 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9dc00126f0 con 0x7f9dd0060200 2024-02-19T22:19:48.424 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:19:48.424 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:48.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:48.422+0000 7f9dd5be4700 1 -- 172.21.15.88:0/3021339757 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f9db802ff10 msgr2=0x7f9db80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:48.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:48.422+0000 7f9dd5be4700 1 --2- 172.21.15.88:0/3021339757 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f9db802ff10 0x7f9db80323d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f9dc40096a0 tx=0x7f9dc4006b40).stop 2024-02-19T22:19:48.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:48.422+0000 7f9dd5be4700 1 -- 172.21.15.88:0/3021339757 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9dd0060200 msgr2=0x7f9dd0106770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:48.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:48.422+0000 7f9dd5be4700 1 --2- 172.21.15.88:0/3021339757 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9dd0060200 0x7f9dd0106770 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f9dc00089f0 tx=0x7f9dc000b040).stop 2024-02-19T22:19:48.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:48.422+0000 7f9dd5be4700 1 -- 172.21.15.88:0/3021339757 shutdown_connections 2024-02-19T22:19:48.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:48.422+0000 7f9dd5be4700 1 --2- 172.21.15.88:0/3021339757 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f9db802ff10 0x7f9db80323d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:48.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:48.422+0000 7f9dd5be4700 1 --2- 172.21.15.88:0/3021339757 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9dd0060200 0x7f9dd0106770 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:48.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:48.422+0000 7f9dd5be4700 1 -- 172.21.15.88:0/3021339757 >> 172.21.15.88:0/3021339757 conn(0x7f9dd00f0b70 msgr2=0x7f9dd00f15e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:48.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:48.422+0000 7f9dd5be4700 1 -- 172.21.15.88:0/3021339757 shutdown_connections 2024-02-19T22:19:48.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:48.423+0000 7f9dd5be4700 1 -- 172.21.15.88:0/3021339757 wait complete. 2024-02-19T22:19:48.426 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:19:49.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:49 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:19:47.714106+0000 mgr.smithi052.ghqrtk (mgr.14162) 311 : cluster [DBG] pgmap v296: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:49.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:49 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:19:48.422080+0000 mon.smithi052 (mon.0) 303 : audit [DBG] from='client.? 172.21.15.88:0/3021339757' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:50.415 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:50.415 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:51.343 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:50 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:19:49.714509+0000 mgr.smithi052.ghqrtk (mgr.14162) 312 : cluster [DBG] pgmap v297: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:51.343 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:50 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:19:49.979999+0000 mon.smithi052 (mon.0) 304 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' 2024-02-19T22:19:51.343 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:50 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:19:50.589100+0000 mon.smithi052 (mon.0) 305 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' 2024-02-19T22:19:51.343 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:50 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:19:50.591956+0000 mon.smithi052 (mon.0) 306 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' 2024-02-19T22:19:52.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:52.312+0000 7eff08163700 1 -- 172.21.15.88:0/130047313 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7eff000f5410 msgr2=0x7eff000f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:52.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:52.312+0000 7eff08163700 1 --2- 172.21.15.88:0/130047313 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7eff000f5410 0x7eff000f5830 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7efef0009fb0 tx=0x7efef0009b30).stop 2024-02-19T22:19:52.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:52.313+0000 7eff08163700 1 -- 172.21.15.88:0/130047313 shutdown_connections 2024-02-19T22:19:52.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:52.313+0000 7eff08163700 1 --2- 172.21.15.88:0/130047313 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7eff000f5410 0x7eff000f5830 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:52.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:52.313+0000 7eff08163700 1 -- 172.21.15.88:0/130047313 >> 172.21.15.88:0/130047313 conn(0x7eff000f0b70 msgr2=0x7eff000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:52.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:52.313+0000 7eff08163700 1 -- 172.21.15.88:0/130047313 shutdown_connections 2024-02-19T22:19:52.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:52.313+0000 7eff08163700 1 -- 172.21.15.88:0/130047313 wait complete. 2024-02-19T22:19:52.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:52.313+0000 7eff08163700 1 Processor -- start 2024-02-19T22:19:52.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:52.313+0000 7eff08163700 1 -- start start 2024-02-19T22:19:52.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:52.313+0000 7eff08163700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7eff000f5410 0x7eff00109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:52.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:52.313+0000 7eff08163700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff00060240 con 0x7eff000f5410 2024-02-19T22:19:52.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:52.313+0000 7eff05eff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7eff000f5410 0x7eff00109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:52.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:52.313+0000 7eff05eff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7eff000f5410 0x7eff00109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:52876/0 (socket says 172.21.15.88:52876) 2024-02-19T22:19:52.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:52.313+0000 7eff05eff700 1 -- 172.21.15.88:0/561542981 learned_addr learned my addr 172.21.15.88:0/561542981 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:19:52.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:52.314+0000 7eff05eff700 1 -- 172.21.15.88:0/561542981 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef0005040 con 0x7eff000f5410 2024-02-19T22:19:52.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:52.314+0000 7eff05eff700 1 --2- 172.21.15.88:0/561542981 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7eff000f5410 0x7eff00109510 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7efef0005010 tx=0x7efef000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:52.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:52.314+0000 7efef67fc700 1 -- 172.21.15.88:0/561542981 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efef0009320 con 0x7eff000f5410 2024-02-19T22:19:52.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:52.314+0000 7eff08163700 1 -- 172.21.15.88:0/561542981 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff00060440 con 0x7eff000f5410 2024-02-19T22:19:52.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:52.314+0000 7efef67fc700 1 -- 172.21.15.88:0/561542981 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efef0013380 con 0x7eff000f5410 2024-02-19T22:19:52.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:52.314+0000 7efef67fc700 1 -- 172.21.15.88:0/561542981 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efef0014e50 con 0x7eff000f5410 2024-02-19T22:19:52.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:52.314+0000 7eff08163700 1 -- 172.21.15.88:0/561542981 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff00109bc0 con 0x7eff000f5410 2024-02-19T22:19:52.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:52.315+0000 7efef67fc700 1 -- 172.21.15.88:0/561542981 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efef00263a0 con 0x7eff000f5410 2024-02-19T22:19:52.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:52.315+0000 7eff08163700 1 -- 172.21.15.88:0/561542981 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efee8004fa0 con 0x7eff000f5410 2024-02-19T22:19:52.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:52.315+0000 7efef67fc700 1 --2- 172.21.15.88:0/561542981 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7efeec02ff10 0x7efeec0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:52.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:52.315+0000 7efef67fc700 1 -- 172.21.15.88:0/561542981 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efef0040eb0 con 0x7eff000f5410 2024-02-19T22:19:52.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:52.315+0000 7eff056fe700 1 --2- 172.21.15.88:0/561542981 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7efeec02ff10 0x7efeec0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:52.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:52.316+0000 7eff056fe700 1 --2- 172.21.15.88:0/561542981 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7efeec02ff10 0x7efeec0323d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7efefc0096a0 tx=0x7efefc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:52.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:52.320+0000 7efef67fc700 1 -- 172.21.15.88:0/561542981 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efef00126f0 con 0x7eff000f5410 2024-02-19T22:19:52.343 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:51 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:19:50.589833+0000 mgr.smithi052.ghqrtk (mgr.14162) 313 : cluster [DBG] pgmap v298: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:52.343 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:51 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cephadm 2024-02-19T22:19:50.593979+0000 mgr.smithi052.ghqrtk (mgr.14162) 314 : cephadm [INF] Deploying daemon alertmanager.smithi088 on smithi088 2024-02-19T22:19:52.343 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:51 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:19:51.587703+0000 mon.smithi052 (mon.0) 307 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-19T22:19:52.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:52.525+0000 7eff08163700 1 -- 172.21.15.88:0/561542981 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efee8005670 con 0x7eff000f5410 2024-02-19T22:19:52.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:52.526+0000 7efef67fc700 1 -- 172.21.15.88:0/561542981 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efef0012330 con 0x7eff000f5410 2024-02-19T22:19:52.528 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:19:52.529 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:52.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:52.528+0000 7eff08163700 1 -- 172.21.15.88:0/561542981 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7efeec02ff10 msgr2=0x7efeec0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:52.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:52.528+0000 7eff08163700 1 --2- 172.21.15.88:0/561542981 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7efeec02ff10 0x7efeec0323d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7efefc0096a0 tx=0x7efefc006b40).stop 2024-02-19T22:19:52.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:52.528+0000 7eff08163700 1 -- 172.21.15.88:0/561542981 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7eff000f5410 msgr2=0x7eff00109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:52.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:52.528+0000 7eff08163700 1 --2- 172.21.15.88:0/561542981 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7eff000f5410 0x7eff00109510 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7efef0005010 tx=0x7efef000b040).stop 2024-02-19T22:19:52.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:52.528+0000 7eff08163700 1 -- 172.21.15.88:0/561542981 shutdown_connections 2024-02-19T22:19:52.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:52.528+0000 7eff08163700 1 --2- 172.21.15.88:0/561542981 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7efeec02ff10 0x7efeec0323d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:52.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:52.528+0000 7eff08163700 1 --2- 172.21.15.88:0/561542981 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7eff000f5410 0x7eff00109510 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:52.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:52.528+0000 7eff08163700 1 -- 172.21.15.88:0/561542981 >> 172.21.15.88:0/561542981 conn(0x7eff000f0b70 msgr2=0x7eff000f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:52.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:52.528+0000 7eff08163700 1 -- 172.21.15.88:0/561542981 shutdown_connections 2024-02-19T22:19:52.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:52.528+0000 7eff08163700 1 -- 172.21.15.88:0/561542981 wait complete. 2024-02-19T22:19:52.531 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:19:53.343 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:52 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:19:52.527395+0000 mon.smithi052 (mon.0) 308 : audit [DBG] from='client.? 172.21.15.88:0/561542981' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:54.305 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:54.305 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:54.343 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:53 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:19:52.590197+0000 mgr.smithi052.ghqrtk (mgr.14162) 315 : cluster [DBG] pgmap v299: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:56.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:56.273+0000 7f266baeb700 1 -- 172.21.15.88:0/3886868537 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f26640f5410 msgr2=0x7f26640f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:56.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:56.273+0000 7f266baeb700 1 --2- 172.21.15.88:0/3886868537 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f26640f5410 0x7f26640f5830 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f2654009fb0 tx=0x7f2654009b30).stop 2024-02-19T22:19:56.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:56.273+0000 7f266baeb700 1 -- 172.21.15.88:0/3886868537 shutdown_connections 2024-02-19T22:19:56.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:56.273+0000 7f266baeb700 1 --2- 172.21.15.88:0/3886868537 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f26640f5410 0x7f26640f5830 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:56.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:56.273+0000 7f266baeb700 1 -- 172.21.15.88:0/3886868537 >> 172.21.15.88:0/3886868537 conn(0x7f26640f0b70 msgr2=0x7f26640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:56.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:56.273+0000 7f266baeb700 1 -- 172.21.15.88:0/3886868537 shutdown_connections 2024-02-19T22:19:56.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:56.273+0000 7f266baeb700 1 -- 172.21.15.88:0/3886868537 wait complete. 2024-02-19T22:19:56.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:56.273+0000 7f266baeb700 1 Processor -- start 2024-02-19T22:19:56.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:56.274+0000 7f266baeb700 1 -- start start 2024-02-19T22:19:56.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:56.274+0000 7f266baeb700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f26640f5410 0x7f26641056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:56.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:56.274+0000 7f266baeb700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2664105bd0 con 0x7f26640f5410 2024-02-19T22:19:56.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:56.274+0000 7f2669887700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f26640f5410 0x7f26641056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:56.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:56.274+0000 7f2669887700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f26640f5410 0x7f26641056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:34258/0 (socket says 172.21.15.88:34258) 2024-02-19T22:19:56.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:56.274+0000 7f2669887700 1 -- 172.21.15.88:0/396811710 learned_addr learned my addr 172.21.15.88:0/396811710 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:19:56.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:56.274+0000 7f2669887700 1 -- 172.21.15.88:0/396811710 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2654005040 con 0x7f26640f5410 2024-02-19T22:19:56.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:56.275+0000 7f2669887700 1 --2- 172.21.15.88:0/396811710 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f26640f5410 0x7f26641056e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f2654005010 tx=0x7f265400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:56.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:56.275+0000 7f265a7fc700 1 -- 172.21.15.88:0/396811710 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2654009320 con 0x7f26640f5410 2024-02-19T22:19:56.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:56.275+0000 7f265a7fc700 1 -- 172.21.15.88:0/396811710 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2654013380 con 0x7f26640f5410 2024-02-19T22:19:56.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:56.275+0000 7f266baeb700 1 -- 172.21.15.88:0/396811710 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2664105dd0 con 0x7f26640f5410 2024-02-19T22:19:56.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:56.275+0000 7f265a7fc700 1 -- 172.21.15.88:0/396811710 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2654014e50 con 0x7f26640f5410 2024-02-19T22:19:56.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:56.275+0000 7f266baeb700 1 -- 172.21.15.88:0/396811710 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2664106260 con 0x7f26640f5410 2024-02-19T22:19:56.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:56.276+0000 7f265a7fc700 1 -- 172.21.15.88:0/396811710 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f26540095f0 con 0x7f26640f5410 2024-02-19T22:19:56.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:56.276+0000 7f266baeb700 1 -- 172.21.15.88:0/396811710 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2664045ad0 con 0x7f26640f5410 2024-02-19T22:19:56.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:56.276+0000 7f265a7fc700 1 --2- 172.21.15.88:0/396811710 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f265002ff10 0x7f26500323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:56.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:56.276+0000 7f265a7fc700 1 -- 172.21.15.88:0/396811710 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2654040e90 con 0x7f26640f5410 2024-02-19T22:19:56.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:56.276+0000 7f2669086700 1 --2- 172.21.15.88:0/396811710 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f265002ff10 0x7f26500323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:56.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:56.277+0000 7f2669086700 1 --2- 172.21.15.88:0/396811710 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f265002ff10 0x7f26500323d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f26600096a0 tx=0x7f2660006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:56.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:56.281+0000 7f265a7fc700 1 -- 172.21.15.88:0/396811710 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2654014420 con 0x7f26640f5410 2024-02-19T22:19:56.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:56.488+0000 7f266baeb700 1 -- 172.21.15.88:0/396811710 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2664107110 con 0x7f26640f5410 2024-02-19T22:19:56.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:56.488+0000 7f265a7fc700 1 -- 172.21.15.88:0/396811710 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2654010230 con 0x7f26640f5410 2024-02-19T22:19:56.491 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:19:56.491 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:56.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:56.490+0000 7f266baeb700 1 -- 172.21.15.88:0/396811710 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f265002ff10 msgr2=0x7f26500323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:56.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:56.490+0000 7f266baeb700 1 --2- 172.21.15.88:0/396811710 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f265002ff10 0x7f26500323d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f26600096a0 tx=0x7f2660006b40).stop 2024-02-19T22:19:56.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:56.490+0000 7f266baeb700 1 -- 172.21.15.88:0/396811710 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f26640f5410 msgr2=0x7f26641056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:56.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:56.490+0000 7f266baeb700 1 --2- 172.21.15.88:0/396811710 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f26640f5410 0x7f26641056e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f2654005010 tx=0x7f265400b040).stop 2024-02-19T22:19:56.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:56.490+0000 7f266baeb700 1 -- 172.21.15.88:0/396811710 shutdown_connections 2024-02-19T22:19:56.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:56.490+0000 7f266baeb700 1 --2- 172.21.15.88:0/396811710 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f265002ff10 0x7f26500323d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:56.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:56.490+0000 7f266baeb700 1 --2- 172.21.15.88:0/396811710 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f26640f5410 0x7f26641056e0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:56.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:56.490+0000 7f266baeb700 1 -- 172.21.15.88:0/396811710 >> 172.21.15.88:0/396811710 conn(0x7f26640f0b70 msgr2=0x7f26640f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:56.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:56.490+0000 7f266baeb700 1 -- 172.21.15.88:0/396811710 shutdown_connections 2024-02-19T22:19:56.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:19:56.490+0000 7f266baeb700 1 -- 172.21.15.88:0/396811710 wait complete. 2024-02-19T22:19:56.494 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:19:56.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:19:54.590566+0000 mgr.smithi052.ghqrtk (mgr.14162) 316 : cluster [DBG] pgmap v300: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:57.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:57 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:19:56.489829+0000 mon.smithi052 (mon.0) 309 : audit [DBG] from='client.? 172.21.15.88:0/396811710' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:58.252 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:58.252 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:58.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:19:58 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:19:56.590848+0000 mgr.smithi052.ghqrtk (mgr.14162) 317 : cluster [DBG] pgmap v301: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:00.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:00.152+0000 7fbc8f31b700 1 -- 172.21.15.88:0/3605560471 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc880f6510 msgr2=0x7fbc880f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:00.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:00.152+0000 7fbc8f31b700 1 --2- 172.21.15.88:0/3605560471 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc880f6510 0x7fbc880f6930 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7fbc78009fb0 tx=0x7fbc78009b30).stop 2024-02-19T22:20:00.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:00.152+0000 7fbc8f31b700 1 -- 172.21.15.88:0/3605560471 shutdown_connections 2024-02-19T22:20:00.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:00.152+0000 7fbc8f31b700 1 --2- 172.21.15.88:0/3605560471 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc880f6510 0x7fbc880f6930 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:00.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:00.152+0000 7fbc8f31b700 1 -- 172.21.15.88:0/3605560471 >> 172.21.15.88:0/3605560471 conn(0x7fbc880f1c70 msgr2=0x7fbc880f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:00.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:00.152+0000 7fbc8f31b700 1 -- 172.21.15.88:0/3605560471 shutdown_connections 2024-02-19T22:20:00.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:00.152+0000 7fbc8f31b700 1 -- 172.21.15.88:0/3605560471 wait complete. 2024-02-19T22:20:00.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:00.153+0000 7fbc8f31b700 1 Processor -- start 2024-02-19T22:20:00.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:00.153+0000 7fbc8f31b700 1 -- start start 2024-02-19T22:20:00.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:00.153+0000 7fbc8f31b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc880f6510 0x7fbc881067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:00.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:00.153+0000 7fbc8f31b700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc88106cd0 con 0x7fbc880f6510 2024-02-19T22:20:00.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:00.153+0000 7fbc8d0b7700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc880f6510 0x7fbc881067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:00.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:00.153+0000 7fbc8d0b7700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc880f6510 0x7fbc881067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:34280/0 (socket says 172.21.15.88:34280) 2024-02-19T22:20:00.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:00.153+0000 7fbc8d0b7700 1 -- 172.21.15.88:0/2003417311 learned_addr learned my addr 172.21.15.88:0/2003417311 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:20:00.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:00.153+0000 7fbc8d0b7700 1 -- 172.21.15.88:0/2003417311 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc78005040 con 0x7fbc880f6510 2024-02-19T22:20:00.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:00.153+0000 7fbc8d0b7700 1 --2- 172.21.15.88:0/2003417311 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc880f6510 0x7fbc881067e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7fbc78005010 tx=0x7fbc7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:00.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:00.154+0000 7fbc7dffb700 1 -- 172.21.15.88:0/2003417311 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc78009320 con 0x7fbc880f6510 2024-02-19T22:20:00.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:00.154+0000 7fbc8f31b700 1 -- 172.21.15.88:0/2003417311 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc88106ed0 con 0x7fbc880f6510 2024-02-19T22:20:00.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:00.154+0000 7fbc7dffb700 1 -- 172.21.15.88:0/2003417311 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbc78013380 con 0x7fbc880f6510 2024-02-19T22:20:00.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:00.154+0000 7fbc7dffb700 1 -- 172.21.15.88:0/2003417311 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc78014e50 con 0x7fbc880f6510 2024-02-19T22:20:00.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:00.154+0000 7fbc8f31b700 1 -- 172.21.15.88:0/2003417311 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc88107360 con 0x7fbc880f6510 2024-02-19T22:20:00.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:00.155+0000 7fbc8f31b700 1 -- 172.21.15.88:0/2003417311 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc88045ad0 con 0x7fbc880f6510 2024-02-19T22:20:00.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:00.155+0000 7fbc7dffb700 1 -- 172.21.15.88:0/2003417311 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbc780095f0 con 0x7fbc880f6510 2024-02-19T22:20:00.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:00.155+0000 7fbc7dffb700 1 --2- 172.21.15.88:0/2003417311 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fbc7402ff10 0x7fbc740323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:00.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:00.155+0000 7fbc7dffb700 1 -- 172.21.15.88:0/2003417311 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbc78040f10 con 0x7fbc880f6510 2024-02-19T22:20:00.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:00.155+0000 7fbc8c8b6700 1 --2- 172.21.15.88:0/2003417311 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fbc7402ff10 0x7fbc740323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:00.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:00.155+0000 7fbc8c8b6700 1 --2- 172.21.15.88:0/2003417311 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fbc7402ff10 0x7fbc740323d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fbc840096a0 tx=0x7fbc84006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:00.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:00.160+0000 7fbc7dffb700 1 -- 172.21.15.88:0/2003417311 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbc78014420 con 0x7fbc880f6510 2024-02-19T22:20:00.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:00.363+0000 7fbc8f31b700 1 -- 172.21.15.88:0/2003417311 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbc881081b0 con 0x7fbc880f6510 2024-02-19T22:20:00.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:00.365+0000 7fbc7dffb700 1 -- 172.21.15.88:0/2003417311 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbc78010230 con 0x7fbc880f6510 2024-02-19T22:20:00.367 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:20:00.367 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:00.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:00.366+0000 7fbc8f31b700 1 -- 172.21.15.88:0/2003417311 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fbc7402ff10 msgr2=0x7fbc740323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:00.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:00.366+0000 7fbc8f31b700 1 --2- 172.21.15.88:0/2003417311 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fbc7402ff10 0x7fbc740323d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fbc840096a0 tx=0x7fbc84006b40).stop 2024-02-19T22:20:00.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:00.366+0000 7fbc8f31b700 1 -- 172.21.15.88:0/2003417311 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc880f6510 msgr2=0x7fbc881067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:00.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:00.366+0000 7fbc8f31b700 1 --2- 172.21.15.88:0/2003417311 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc880f6510 0x7fbc881067e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7fbc78005010 tx=0x7fbc7800b040).stop 2024-02-19T22:20:00.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:00.366+0000 7fbc8f31b700 1 -- 172.21.15.88:0/2003417311 shutdown_connections 2024-02-19T22:20:00.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:00.366+0000 7fbc8f31b700 1 --2- 172.21.15.88:0/2003417311 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fbc7402ff10 0x7fbc740323d0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:00.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:00.366+0000 7fbc8f31b700 1 --2- 172.21.15.88:0/2003417311 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc880f6510 0x7fbc881067e0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:00.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:00.366+0000 7fbc8f31b700 1 -- 172.21.15.88:0/2003417311 >> 172.21.15.88:0/2003417311 conn(0x7fbc880f1c70 msgr2=0x7fbc880f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:00.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:00.366+0000 7fbc8f31b700 1 -- 172.21.15.88:0/2003417311 shutdown_connections 2024-02-19T22:20:00.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:00.366+0000 7fbc8f31b700 1 -- 172.21.15.88:0/2003417311 wait complete. 2024-02-19T22:20:00.369 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:20:00.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:00 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:19:58.591253+0000 mgr.smithi052.ghqrtk (mgr.14162) 318 : cluster [DBG] pgmap v302: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:01.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:01 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:20:00.365300+0000 mon.smithi052 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.88:0/2003417311' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:02.008 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:02.009 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:02.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:02 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:20:00.591656+0000 mgr.smithi052.ghqrtk (mgr.14162) 319 : cluster [DBG] pgmap v303: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:03.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:03.863+0000 7f47433fc700 1 -- 172.21.15.88:0/3801905583 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f473c0f6510 msgr2=0x7f473c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:03.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:03.863+0000 7f47433fc700 1 --2- 172.21.15.88:0/3801905583 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f473c0f6510 0x7f473c0f6930 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f472c009fb0 tx=0x7f472c009b30).stop 2024-02-19T22:20:03.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:03.863+0000 7f47433fc700 1 -- 172.21.15.88:0/3801905583 shutdown_connections 2024-02-19T22:20:03.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:03.863+0000 7f47433fc700 1 --2- 172.21.15.88:0/3801905583 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f473c0f6510 0x7f473c0f6930 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:03.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:03.863+0000 7f47433fc700 1 -- 172.21.15.88:0/3801905583 >> 172.21.15.88:0/3801905583 conn(0x7f473c0f1c70 msgr2=0x7f473c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:03.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:03.863+0000 7f47433fc700 1 -- 172.21.15.88:0/3801905583 shutdown_connections 2024-02-19T22:20:03.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:03.863+0000 7f47433fc700 1 -- 172.21.15.88:0/3801905583 wait complete. 2024-02-19T22:20:03.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:03.864+0000 7f47433fc700 1 Processor -- start 2024-02-19T22:20:03.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:03.864+0000 7f47433fc700 1 -- start start 2024-02-19T22:20:03.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:03.864+0000 7f47433fc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f473c0f6510 0x7f473c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:03.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:03.864+0000 7f47433fc700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f473c069e20 con 0x7f473c0f6510 2024-02-19T22:20:03.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:03.864+0000 7f4741198700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f473c0f6510 0x7f473c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:03.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:03.864+0000 7f4741198700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f473c0f6510 0x7f473c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:59208/0 (socket says 172.21.15.88:59208) 2024-02-19T22:20:03.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:03.864+0000 7f4741198700 1 -- 172.21.15.88:0/2094632694 learned_addr learned my addr 172.21.15.88:0/2094632694 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:20:03.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:03.865+0000 7f4741198700 1 -- 172.21.15.88:0/2094632694 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f472c005040 con 0x7f473c0f6510 2024-02-19T22:20:03.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:03.865+0000 7f4741198700 1 --2- 172.21.15.88:0/2094632694 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f473c0f6510 0x7f473c06b9a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f472c005010 tx=0x7f472c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:03.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:03.865+0000 7f4731ffb700 1 -- 172.21.15.88:0/2094632694 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f472c009320 con 0x7f473c0f6510 2024-02-19T22:20:03.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:03.865+0000 7f47433fc700 1 -- 172.21.15.88:0/2094632694 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f473c069fc0 con 0x7f473c0f6510 2024-02-19T22:20:03.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:03.865+0000 7f4731ffb700 1 -- 172.21.15.88:0/2094632694 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f472c013380 con 0x7f473c0f6510 2024-02-19T22:20:03.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:03.865+0000 7f4731ffb700 1 -- 172.21.15.88:0/2094632694 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f472c014e50 con 0x7f473c0f6510 2024-02-19T22:20:03.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:03.865+0000 7f47433fc700 1 -- 172.21.15.88:0/2094632694 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f473c06a450 con 0x7f473c0f6510 2024-02-19T22:20:03.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:03.866+0000 7f4731ffb700 1 -- 172.21.15.88:0/2094632694 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f472c0095f0 con 0x7f473c0f6510 2024-02-19T22:20:03.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:03.866+0000 7f47433fc700 1 -- 172.21.15.88:0/2094632694 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f473c059100 con 0x7f473c0f6510 2024-02-19T22:20:03.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:03.866+0000 7f4731ffb700 1 --2- 172.21.15.88:0/2094632694 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f472802ff10 0x7f47280323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:03.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:03.866+0000 7f4731ffb700 1 -- 172.21.15.88:0/2094632694 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f472c040ee0 con 0x7f473c0f6510 2024-02-19T22:20:03.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:03.866+0000 7f4740997700 1 --2- 172.21.15.88:0/2094632694 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f472802ff10 0x7f47280323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:03.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:03.867+0000 7f4740997700 1 --2- 172.21.15.88:0/2094632694 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f472802ff10 0x7f47280323d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f47380096a0 tx=0x7f4738006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:03.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:03.871+0000 7f4731ffb700 1 -- 172.21.15.88:0/2094632694 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f472c014420 con 0x7f473c0f6510 2024-02-19T22:20:04.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:04.072+0000 7f47433fc700 1 -- 172.21.15.88:0/2094632694 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f473c0fc0a0 con 0x7f473c0f6510 2024-02-19T22:20:04.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:04.072+0000 7f4731ffb700 1 -- 172.21.15.88:0/2094632694 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f472c01d980 con 0x7f473c0f6510 2024-02-19T22:20:04.075 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:20:04.075 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:04.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:04.074+0000 7f47433fc700 1 -- 172.21.15.88:0/2094632694 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f472802ff10 msgr2=0x7f47280323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:04.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:04.074+0000 7f47433fc700 1 --2- 172.21.15.88:0/2094632694 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f472802ff10 0x7f47280323d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f47380096a0 tx=0x7f4738006b40).stop 2024-02-19T22:20:04.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:04.074+0000 7f47433fc700 1 -- 172.21.15.88:0/2094632694 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f473c0f6510 msgr2=0x7f473c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:04.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:04.074+0000 7f47433fc700 1 --2- 172.21.15.88:0/2094632694 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f473c0f6510 0x7f473c06b9a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f472c005010 tx=0x7f472c00b040).stop 2024-02-19T22:20:04.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:04.074+0000 7f47433fc700 1 -- 172.21.15.88:0/2094632694 shutdown_connections 2024-02-19T22:20:04.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:04.074+0000 7f47433fc700 1 --2- 172.21.15.88:0/2094632694 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f472802ff10 0x7f47280323d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:04.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:04.074+0000 7f47433fc700 1 --2- 172.21.15.88:0/2094632694 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f473c0f6510 0x7f473c06b9a0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:04.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:04.074+0000 7f47433fc700 1 -- 172.21.15.88:0/2094632694 >> 172.21.15.88:0/2094632694 conn(0x7f473c0f1c70 msgr2=0x7f473c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:04.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:04.074+0000 7f47433fc700 1 -- 172.21.15.88:0/2094632694 shutdown_connections 2024-02-19T22:20:04.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:04.074+0000 7f47433fc700 1 -- 172.21.15.88:0/2094632694 wait complete. 2024-02-19T22:20:04.078 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:20:04.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:04 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:20:02.592029+0000 mgr.smithi052.ghqrtk (mgr.14162) 320 : cluster [DBG] pgmap v304: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:04.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:04 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:20:04.074017+0000 mon.smithi052 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.88:0/2094632694' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:05.688 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:05.688 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:06.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:06 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:20:04.592355+0000 mgr.smithi052.ghqrtk (mgr.14162) 321 : cluster [DBG] pgmap v305: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:07.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:07.768+0000 7fccaf24b700 1 -- 172.21.15.88:0/3985202131 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcca80f6510 msgr2=0x7fcca80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:07.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:07.768+0000 7fccaf24b700 1 --2- 172.21.15.88:0/3985202131 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcca80f6510 0x7fcca80f6930 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7fcc9c009fb0 tx=0x7fcc9c009b30).stop 2024-02-19T22:20:07.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:07.768+0000 7fccaf24b700 1 -- 172.21.15.88:0/3985202131 shutdown_connections 2024-02-19T22:20:07.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:07.768+0000 7fccaf24b700 1 --2- 172.21.15.88:0/3985202131 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcca80f6510 0x7fcca80f6930 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:07.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:07.768+0000 7fccaf24b700 1 -- 172.21.15.88:0/3985202131 >> 172.21.15.88:0/3985202131 conn(0x7fcca80f1c70 msgr2=0x7fcca80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:07.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:07.768+0000 7fccaf24b700 1 -- 172.21.15.88:0/3985202131 shutdown_connections 2024-02-19T22:20:07.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:07.768+0000 7fccaf24b700 1 -- 172.21.15.88:0/3985202131 wait complete. 2024-02-19T22:20:07.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:07.768+0000 7fccaf24b700 1 Processor -- start 2024-02-19T22:20:07.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:07.768+0000 7fccaf24b700 1 -- start start 2024-02-19T22:20:07.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:07.769+0000 7fccaf24b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcca80f6510 0x7fcca81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:07.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:07.769+0000 7fccaf24b700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcca8106cd0 con 0x7fcca80f6510 2024-02-19T22:20:07.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:07.769+0000 7fccacfe7700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcca80f6510 0x7fcca81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:07.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:07.769+0000 7fccacfe7700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcca80f6510 0x7fcca81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:59232/0 (socket says 172.21.15.88:59232) 2024-02-19T22:20:07.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:07.769+0000 7fccacfe7700 1 -- 172.21.15.88:0/4069330061 learned_addr learned my addr 172.21.15.88:0/4069330061 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:20:07.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:07.769+0000 7fccacfe7700 1 -- 172.21.15.88:0/4069330061 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc9c005040 con 0x7fcca80f6510 2024-02-19T22:20:07.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:07.769+0000 7fccacfe7700 1 --2- 172.21.15.88:0/4069330061 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcca80f6510 0x7fcca81067e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7fcc9c005010 tx=0x7fcc9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:07.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:07.769+0000 7fcca57fa700 1 -- 172.21.15.88:0/4069330061 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc9c009320 con 0x7fcca80f6510 2024-02-19T22:20:07.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:07.770+0000 7fccaf24b700 1 -- 172.21.15.88:0/4069330061 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcca8106ed0 con 0x7fcca80f6510 2024-02-19T22:20:07.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:07.770+0000 7fcca57fa700 1 -- 172.21.15.88:0/4069330061 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcc9c013380 con 0x7fcca80f6510 2024-02-19T22:20:07.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:07.770+0000 7fcca57fa700 1 -- 172.21.15.88:0/4069330061 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc9c014e50 con 0x7fcca80f6510 2024-02-19T22:20:07.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:07.770+0000 7fccaf24b700 1 -- 172.21.15.88:0/4069330061 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcca8107360 con 0x7fcca80f6510 2024-02-19T22:20:07.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:07.770+0000 7fcca57fa700 1 -- 172.21.15.88:0/4069330061 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcc9c014420 con 0x7fcca80f6510 2024-02-19T22:20:07.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:07.770+0000 7fccaf24b700 1 -- 172.21.15.88:0/4069330061 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcca8045ad0 con 0x7fcca80f6510 2024-02-19T22:20:07.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:07.771+0000 7fcca57fa700 1 --2- 172.21.15.88:0/4069330061 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fcc9802ff10 0x7fcc980323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:07.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:07.771+0000 7fcca57fa700 1 -- 172.21.15.88:0/4069330061 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcc9c040f40 con 0x7fcca80f6510 2024-02-19T22:20:07.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:07.771+0000 7fcca7fff700 1 --2- 172.21.15.88:0/4069330061 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fcc9802ff10 0x7fcc980323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:07.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:07.771+0000 7fcca7fff700 1 --2- 172.21.15.88:0/4069330061 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fcc9802ff10 0x7fcc980323d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fcc940096a0 tx=0x7fcc94006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:07.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:07.776+0000 7fcca57fa700 1 -- 172.21.15.88:0/4069330061 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcc9c0095f0 con 0x7fcca80f6510 2024-02-19T22:20:07.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:07.989+0000 7fccaf24b700 1 -- 172.21.15.88:0/4069330061 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcca8108210 con 0x7fcca80f6510 2024-02-19T22:20:07.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:07.989+0000 7fcca57fa700 1 -- 172.21.15.88:0/4069330061 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcc9c01db70 con 0x7fcca80f6510 2024-02-19T22:20:07.992 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:20:07.992 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:07.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:07.991+0000 7fccaf24b700 1 -- 172.21.15.88:0/4069330061 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fcc9802ff10 msgr2=0x7fcc980323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:07.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:07.991+0000 7fccaf24b700 1 --2- 172.21.15.88:0/4069330061 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fcc9802ff10 0x7fcc980323d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fcc940096a0 tx=0x7fcc94006b40).stop 2024-02-19T22:20:07.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:07.991+0000 7fccaf24b700 1 -- 172.21.15.88:0/4069330061 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcca80f6510 msgr2=0x7fcca81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:07.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:07.991+0000 7fccaf24b700 1 --2- 172.21.15.88:0/4069330061 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcca80f6510 0x7fcca81067e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7fcc9c005010 tx=0x7fcc9c00b040).stop 2024-02-19T22:20:07.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:07.991+0000 7fccaf24b700 1 -- 172.21.15.88:0/4069330061 shutdown_connections 2024-02-19T22:20:07.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:07.991+0000 7fccaf24b700 1 --2- 172.21.15.88:0/4069330061 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fcc9802ff10 0x7fcc980323d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:07.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:07.991+0000 7fccaf24b700 1 --2- 172.21.15.88:0/4069330061 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcca80f6510 0x7fcca81067e0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:07.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:07.991+0000 7fccaf24b700 1 -- 172.21.15.88:0/4069330061 >> 172.21.15.88:0/4069330061 conn(0x7fcca80f1c70 msgr2=0x7fcca80f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:07.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:07.991+0000 7fccaf24b700 1 -- 172.21.15.88:0/4069330061 shutdown_connections 2024-02-19T22:20:07.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:07.991+0000 7fccaf24b700 1 -- 172.21.15.88:0/4069330061 wait complete. 2024-02-19T22:20:07.995 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:20:08.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:08 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:20:06.592579+0000 mgr.smithi052.ghqrtk (mgr.14162) 322 : cluster [DBG] pgmap v306: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:08.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:08 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:20:07.990968+0000 mon.smithi052 (mon.0) 312 : audit [DBG] from='client.? 172.21.15.88:0/4069330061' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:09.636 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:09.636 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:10.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:10 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:20:08.592824+0000 mgr.smithi052.ghqrtk (mgr.14162) 323 : cluster [DBG] pgmap v307: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:11.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:11.431+0000 7fc3aff8f700 1 -- 172.21.15.88:0/912879794 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc3a80f5410 msgr2=0x7fc3a80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:11.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:11.431+0000 7fc3aff8f700 1 --2- 172.21.15.88:0/912879794 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc3a80f5410 0x7fc3a80f5830 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7fc398009fb0 tx=0x7fc398009b30).stop 2024-02-19T22:20:11.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:11.431+0000 7fc3aff8f700 1 -- 172.21.15.88:0/912879794 shutdown_connections 2024-02-19T22:20:11.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:11.431+0000 7fc3aff8f700 1 --2- 172.21.15.88:0/912879794 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc3a80f5410 0x7fc3a80f5830 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:11.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:11.431+0000 7fc3aff8f700 1 -- 172.21.15.88:0/912879794 >> 172.21.15.88:0/912879794 conn(0x7fc3a80f0b70 msgr2=0x7fc3a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:11.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:11.431+0000 7fc3aff8f700 1 -- 172.21.15.88:0/912879794 shutdown_connections 2024-02-19T22:20:11.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:11.431+0000 7fc3aff8f700 1 -- 172.21.15.88:0/912879794 wait complete. 2024-02-19T22:20:11.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:11.432+0000 7fc3aff8f700 1 Processor -- start 2024-02-19T22:20:11.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:11.432+0000 7fc3aff8f700 1 -- start start 2024-02-19T22:20:11.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:11.432+0000 7fc3aff8f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc3a80f5410 0x7fc3a81094a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:11.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:11.432+0000 7fc3aff8f700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a8060280 con 0x7fc3a80f5410 2024-02-19T22:20:11.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:11.432+0000 7fc3add2b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc3a80f5410 0x7fc3a81094a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:11.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:11.432+0000 7fc3add2b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc3a80f5410 0x7fc3a81094a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:59254/0 (socket says 172.21.15.88:59254) 2024-02-19T22:20:11.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:11.432+0000 7fc3add2b700 1 -- 172.21.15.88:0/1296843033 learned_addr learned my addr 172.21.15.88:0/1296843033 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:20:11.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:11.433+0000 7fc3add2b700 1 -- 172.21.15.88:0/1296843033 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc398005040 con 0x7fc3a80f5410 2024-02-19T22:20:11.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:11.433+0000 7fc3add2b700 1 --2- 172.21.15.88:0/1296843033 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc3a80f5410 0x7fc3a81094a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7fc398005010 tx=0x7fc39800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:11.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:11.433+0000 7fc39e7fc700 1 -- 172.21.15.88:0/1296843033 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc398009320 con 0x7fc3a80f5410 2024-02-19T22:20:11.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:11.433+0000 7fc3aff8f700 1 -- 172.21.15.88:0/1296843033 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3a8060480 con 0x7fc3a80f5410 2024-02-19T22:20:11.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:11.433+0000 7fc39e7fc700 1 -- 172.21.15.88:0/1296843033 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc398013380 con 0x7fc3a80f5410 2024-02-19T22:20:11.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:11.433+0000 7fc39e7fc700 1 -- 172.21.15.88:0/1296843033 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc398014e50 con 0x7fc3a80f5410 2024-02-19T22:20:11.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:11.433+0000 7fc3aff8f700 1 -- 172.21.15.88:0/1296843033 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3a8109bd0 con 0x7fc3a80f5410 2024-02-19T22:20:11.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:11.434+0000 7fc39e7fc700 1 -- 172.21.15.88:0/1296843033 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc3980095f0 con 0x7fc3a80f5410 2024-02-19T22:20:11.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:11.434+0000 7fc3aff8f700 1 -- 172.21.15.88:0/1296843033 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3a8045ad0 con 0x7fc3a80f5410 2024-02-19T22:20:11.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:11.434+0000 7fc39e7fc700 1 --2- 172.21.15.88:0/1296843033 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fc39402ff10 0x7fc3940323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:11.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:11.434+0000 7fc39e7fc700 1 -- 172.21.15.88:0/1296843033 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc398040e90 con 0x7fc3a80f5410 2024-02-19T22:20:11.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:11.434+0000 7fc3ad52a700 1 --2- 172.21.15.88:0/1296843033 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fc39402ff10 0x7fc3940323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:11.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:11.435+0000 7fc3ad52a700 1 --2- 172.21.15.88:0/1296843033 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fc39402ff10 0x7fc3940323d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fc3a40096a0 tx=0x7fc3a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:11.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:11.439+0000 7fc39e7fc700 1 -- 172.21.15.88:0/1296843033 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc398014420 con 0x7fc3a80f5410 2024-02-19T22:20:11.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:11.643+0000 7fc3aff8f700 1 -- 172.21.15.88:0/1296843033 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc3a80fafa0 con 0x7fc3a80f5410 2024-02-19T22:20:11.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:11.643+0000 7fc39e7fc700 1 -- 172.21.15.88:0/1296843033 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc398010230 con 0x7fc3a80f5410 2024-02-19T22:20:11.645 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:20:11.645 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:11.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:11.645+0000 7fc3aff8f700 1 -- 172.21.15.88:0/1296843033 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fc39402ff10 msgr2=0x7fc3940323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:11.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:11.645+0000 7fc3aff8f700 1 --2- 172.21.15.88:0/1296843033 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fc39402ff10 0x7fc3940323d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fc3a40096a0 tx=0x7fc3a4006b40).stop 2024-02-19T22:20:11.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:11.645+0000 7fc3aff8f700 1 -- 172.21.15.88:0/1296843033 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc3a80f5410 msgr2=0x7fc3a81094a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:11.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:11.645+0000 7fc3aff8f700 1 --2- 172.21.15.88:0/1296843033 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc3a80f5410 0x7fc3a81094a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7fc398005010 tx=0x7fc39800b040).stop 2024-02-19T22:20:11.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:11.645+0000 7fc3aff8f700 1 -- 172.21.15.88:0/1296843033 shutdown_connections 2024-02-19T22:20:11.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:11.645+0000 7fc3aff8f700 1 --2- 172.21.15.88:0/1296843033 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fc39402ff10 0x7fc3940323d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:11.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:11.645+0000 7fc3aff8f700 1 --2- 172.21.15.88:0/1296843033 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc3a80f5410 0x7fc3a81094a0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:11.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:11.645+0000 7fc3aff8f700 1 -- 172.21.15.88:0/1296843033 >> 172.21.15.88:0/1296843033 conn(0x7fc3a80f0b70 msgr2=0x7fc3a80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:11.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:11.645+0000 7fc3aff8f700 1 -- 172.21.15.88:0/1296843033 shutdown_connections 2024-02-19T22:20:11.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:11.645+0000 7fc3aff8f700 1 -- 172.21.15.88:0/1296843033 wait complete. 2024-02-19T22:20:11.647 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:20:12.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:12 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:20:10.593095+0000 mgr.smithi052.ghqrtk (mgr.14162) 324 : cluster [DBG] pgmap v308: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:12.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:12 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:20:11.643697+0000 mon.smithi052 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.88:0/1296843033' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:13.388 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:13.388 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:14.551 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:14 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:20:12.593477+0000 mgr.smithi052.ghqrtk (mgr.14162) 325 : cluster [DBG] pgmap v309: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:15.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:15.243+0000 7f0e4ad5a700 1 -- 172.21.15.88:0/2469786083 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0e440f5410 msgr2=0x7f0e440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:15.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:15.243+0000 7f0e4ad5a700 1 --2- 172.21.15.88:0/2469786083 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0e440f5410 0x7f0e440f5830 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f0e38009fb0 tx=0x7f0e38009b30).stop 2024-02-19T22:20:15.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:15.243+0000 7f0e4ad5a700 1 -- 172.21.15.88:0/2469786083 shutdown_connections 2024-02-19T22:20:15.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:15.243+0000 7f0e4ad5a700 1 --2- 172.21.15.88:0/2469786083 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0e440f5410 0x7f0e440f5830 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:15.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:15.243+0000 7f0e4ad5a700 1 -- 172.21.15.88:0/2469786083 >> 172.21.15.88:0/2469786083 conn(0x7f0e440f0b70 msgr2=0x7f0e440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:15.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:15.243+0000 7f0e4ad5a700 1 -- 172.21.15.88:0/2469786083 shutdown_connections 2024-02-19T22:20:15.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:15.243+0000 7f0e4ad5a700 1 -- 172.21.15.88:0/2469786083 wait complete. 2024-02-19T22:20:15.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:15.243+0000 7f0e4ad5a700 1 Processor -- start 2024-02-19T22:20:15.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:15.243+0000 7f0e4ad5a700 1 -- start start 2024-02-19T22:20:15.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:15.244+0000 7f0e4ad5a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0e440f5410 0x7f0e4410b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:15.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:15.244+0000 7f0e4ad5a700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e441072f0 con 0x7f0e440f5410 2024-02-19T22:20:15.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:15.244+0000 7f0e48af6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0e440f5410 0x7f0e4410b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:15.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:15.244+0000 7f0e48af6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0e440f5410 0x7f0e4410b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:53848/0 (socket says 172.21.15.88:53848) 2024-02-19T22:20:15.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:15.244+0000 7f0e48af6700 1 -- 172.21.15.88:0/3717741113 learned_addr learned my addr 172.21.15.88:0/3717741113 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:20:15.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:15.244+0000 7f0e48af6700 1 -- 172.21.15.88:0/3717741113 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e38005040 con 0x7f0e440f5410 2024-02-19T22:20:15.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:15.244+0000 7f0e48af6700 1 --2- 172.21.15.88:0/3717741113 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0e440f5410 0x7f0e4410b900 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f0e38005010 tx=0x7f0e3800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:15.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:15.244+0000 7f0e417fa700 1 -- 172.21.15.88:0/3717741113 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e38009320 con 0x7f0e440f5410 2024-02-19T22:20:15.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:15.244+0000 7f0e4ad5a700 1 -- 172.21.15.88:0/3717741113 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e44060220 con 0x7f0e440f5410 2024-02-19T22:20:15.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:15.244+0000 7f0e417fa700 1 -- 172.21.15.88:0/3717741113 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0e38013380 con 0x7f0e440f5410 2024-02-19T22:20:15.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:15.244+0000 7f0e417fa700 1 -- 172.21.15.88:0/3717741113 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e38014e50 con 0x7f0e440f5410 2024-02-19T22:20:15.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:15.244+0000 7f0e4ad5a700 1 -- 172.21.15.88:0/3717741113 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e4410bdf0 con 0x7f0e440f5410 2024-02-19T22:20:15.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:15.245+0000 7f0e417fa700 1 -- 172.21.15.88:0/3717741113 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0e380095f0 con 0x7f0e440f5410 2024-02-19T22:20:15.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:15.245+0000 7f0e4ad5a700 1 -- 172.21.15.88:0/3717741113 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e44045ad0 con 0x7f0e440f5410 2024-02-19T22:20:15.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:15.245+0000 7f0e417fa700 1 --2- 172.21.15.88:0/3717741113 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f0e3402ff10 0x7f0e340323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:15.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:15.245+0000 7f0e417fa700 1 -- 172.21.15.88:0/3717741113 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0e38040fa0 con 0x7f0e440f5410 2024-02-19T22:20:15.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:15.246+0000 7f0e43fff700 1 --2- 172.21.15.88:0/3717741113 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f0e3402ff10 0x7f0e340323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:15.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:15.246+0000 7f0e43fff700 1 --2- 172.21.15.88:0/3717741113 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f0e3402ff10 0x7f0e340323d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f0e300096a0 tx=0x7f0e30006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:15.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:15.250+0000 7f0e417fa700 1 -- 172.21.15.88:0/3717741113 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0e38014420 con 0x7f0e440f5410 2024-02-19T22:20:15.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:15.453+0000 7f0e4ad5a700 1 -- 172.21.15.88:0/3717741113 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0e4410c5b0 con 0x7f0e440f5410 2024-02-19T22:20:15.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:15.454+0000 7f0e417fa700 1 -- 172.21.15.88:0/3717741113 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0e38010230 con 0x7f0e440f5410 2024-02-19T22:20:15.456 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:20:15.456 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:15.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:15.455+0000 7f0e4ad5a700 1 -- 172.21.15.88:0/3717741113 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f0e3402ff10 msgr2=0x7f0e340323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:15.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:15.455+0000 7f0e4ad5a700 1 --2- 172.21.15.88:0/3717741113 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f0e3402ff10 0x7f0e340323d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f0e300096a0 tx=0x7f0e30006b40).stop 2024-02-19T22:20:15.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:15.455+0000 7f0e4ad5a700 1 -- 172.21.15.88:0/3717741113 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0e440f5410 msgr2=0x7f0e4410b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:15.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:15.456+0000 7f0e4ad5a700 1 --2- 172.21.15.88:0/3717741113 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0e440f5410 0x7f0e4410b900 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f0e38005010 tx=0x7f0e3800b040).stop 2024-02-19T22:20:15.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:15.456+0000 7f0e4ad5a700 1 -- 172.21.15.88:0/3717741113 shutdown_connections 2024-02-19T22:20:15.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:15.456+0000 7f0e4ad5a700 1 --2- 172.21.15.88:0/3717741113 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f0e3402ff10 0x7f0e340323d0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:15.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:15.456+0000 7f0e4ad5a700 1 --2- 172.21.15.88:0/3717741113 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0e440f5410 0x7f0e4410b900 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:15.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:15.456+0000 7f0e4ad5a700 1 -- 172.21.15.88:0/3717741113 >> 172.21.15.88:0/3717741113 conn(0x7f0e440f0b70 msgr2=0x7f0e440f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:15.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:15.456+0000 7f0e4ad5a700 1 -- 172.21.15.88:0/3717741113 shutdown_connections 2024-02-19T22:20:15.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:15.456+0000 7f0e4ad5a700 1 -- 172.21.15.88:0/3717741113 wait complete. 2024-02-19T22:20:15.458 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:20:16.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:16 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:20:14.593825+0000 mgr.smithi052.ghqrtk (mgr.14162) 326 : cluster [DBG] pgmap v310: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:16.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:16 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:20:15.454557+0000 mon.smithi052 (mon.0) 314 : audit [DBG] from='client.? 172.21.15.88:0/3717741113' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:17.232 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:17.232 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:18.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:20:16.594196+0000 mgr.smithi052.ghqrtk (mgr.14162) 327 : cluster [DBG] pgmap v311: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:18.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:18.979+0000 7feb38c9e700 1 -- 172.21.15.88:0/1814031400 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feb340f5410 msgr2=0x7feb340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:18.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:18.979+0000 7feb38c9e700 1 --2- 172.21.15.88:0/1814031400 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feb340f5410 0x7feb340f5830 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7feb24009fb0 tx=0x7feb24009b30).stop 2024-02-19T22:20:18.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:18.979+0000 7feb38c9e700 1 -- 172.21.15.88:0/1814031400 shutdown_connections 2024-02-19T22:20:18.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:18.979+0000 7feb38c9e700 1 --2- 172.21.15.88:0/1814031400 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feb340f5410 0x7feb340f5830 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:18.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:18.979+0000 7feb38c9e700 1 -- 172.21.15.88:0/1814031400 >> 172.21.15.88:0/1814031400 conn(0x7feb340f0b70 msgr2=0x7feb340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:18.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:18.979+0000 7feb38c9e700 1 -- 172.21.15.88:0/1814031400 shutdown_connections 2024-02-19T22:20:18.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:18.979+0000 7feb38c9e700 1 -- 172.21.15.88:0/1814031400 wait complete. 2024-02-19T22:20:18.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:18.980+0000 7feb38c9e700 1 Processor -- start 2024-02-19T22:20:18.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:18.980+0000 7feb38c9e700 1 -- start start 2024-02-19T22:20:18.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:18.980+0000 7feb38c9e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feb340f5410 0x7feb340fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:18.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:18.980+0000 7feb38c9e700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb340fb960 con 0x7feb340f5410 2024-02-19T22:20:18.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:18.980+0000 7feb3259c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feb340f5410 0x7feb340fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:18.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:18.980+0000 7feb3259c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feb340f5410 0x7feb340fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:53870/0 (socket says 172.21.15.88:53870) 2024-02-19T22:20:18.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:18.980+0000 7feb3259c700 1 -- 172.21.15.88:0/492326378 learned_addr learned my addr 172.21.15.88:0/492326378 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:20:18.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:18.981+0000 7feb3259c700 1 -- 172.21.15.88:0/492326378 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb24005040 con 0x7feb340f5410 2024-02-19T22:20:18.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:18.981+0000 7feb3259c700 1 --2- 172.21.15.88:0/492326378 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feb340f5410 0x7feb340fd4e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7feb24005010 tx=0x7feb2400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:18.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:18.981+0000 7feb22ffd700 1 -- 172.21.15.88:0/492326378 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb24009320 con 0x7feb340f5410 2024-02-19T22:20:18.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:18.981+0000 7feb22ffd700 1 -- 172.21.15.88:0/492326378 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feb24013380 con 0x7feb340f5410 2024-02-19T22:20:18.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:18.981+0000 7feb38c9e700 1 -- 172.21.15.88:0/492326378 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb340fbb00 con 0x7feb340f5410 2024-02-19T22:20:18.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:18.981+0000 7feb22ffd700 1 -- 172.21.15.88:0/492326378 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb24014e50 con 0x7feb340f5410 2024-02-19T22:20:18.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:18.981+0000 7feb38c9e700 1 -- 172.21.15.88:0/492326378 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb340fbf90 con 0x7feb340f5410 2024-02-19T22:20:18.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:18.982+0000 7feb22ffd700 1 -- 172.21.15.88:0/492326378 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feb240263a0 con 0x7feb340f5410 2024-02-19T22:20:18.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:18.982+0000 7feb22ffd700 1 --2- 172.21.15.88:0/492326378 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7feb1c02ff10 0x7feb1c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:18.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:18.982+0000 7feb22ffd700 1 -- 172.21.15.88:0/492326378 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feb24040ea0 con 0x7feb340f5410 2024-02-19T22:20:18.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:18.982+0000 7feb38c9e700 1 -- 172.21.15.88:0/492326378 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb34059100 con 0x7feb340f5410 2024-02-19T22:20:18.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:18.982+0000 7feb31d9b700 1 --2- 172.21.15.88:0/492326378 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7feb1c02ff10 0x7feb1c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:18.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:18.983+0000 7feb31d9b700 1 --2- 172.21.15.88:0/492326378 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7feb1c02ff10 0x7feb1c0323d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7feb280096a0 tx=0x7feb28006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:18.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:18.987+0000 7feb22ffd700 1 -- 172.21.15.88:0/492326378 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feb24012920 con 0x7feb340f5410 2024-02-19T22:20:19.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:19.190+0000 7feb38c9e700 1 -- 172.21.15.88:0/492326378 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feb340fafa0 con 0x7feb340f5410 2024-02-19T22:20:19.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:19.191+0000 7feb22ffd700 1 -- 172.21.15.88:0/492326378 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7feb240126f0 con 0x7feb340f5410 2024-02-19T22:20:19.192 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:20:19.192 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:19.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:19.192+0000 7feb38c9e700 1 -- 172.21.15.88:0/492326378 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7feb1c02ff10 msgr2=0x7feb1c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:19.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:19.192+0000 7feb38c9e700 1 --2- 172.21.15.88:0/492326378 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7feb1c02ff10 0x7feb1c0323d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7feb280096a0 tx=0x7feb28006b40).stop 2024-02-19T22:20:19.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:19.192+0000 7feb38c9e700 1 -- 172.21.15.88:0/492326378 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feb340f5410 msgr2=0x7feb340fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:19.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:19.192+0000 7feb38c9e700 1 --2- 172.21.15.88:0/492326378 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feb340f5410 0x7feb340fd4e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7feb24005010 tx=0x7feb2400b040).stop 2024-02-19T22:20:19.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:19.192+0000 7feb38c9e700 1 -- 172.21.15.88:0/492326378 shutdown_connections 2024-02-19T22:20:19.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:19.192+0000 7feb38c9e700 1 --2- 172.21.15.88:0/492326378 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7feb1c02ff10 0x7feb1c0323d0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:19.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:19.192+0000 7feb38c9e700 1 --2- 172.21.15.88:0/492326378 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feb340f5410 0x7feb340fd4e0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:19.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:19.192+0000 7feb38c9e700 1 -- 172.21.15.88:0/492326378 >> 172.21.15.88:0/492326378 conn(0x7feb340f0b70 msgr2=0x7feb3405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:19.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:19.192+0000 7feb38c9e700 1 -- 172.21.15.88:0/492326378 shutdown_connections 2024-02-19T22:20:19.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:19.192+0000 7feb38c9e700 1 -- 172.21.15.88:0/492326378 wait complete. 2024-02-19T22:20:19.195 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:20:20.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:20 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:20:18.594508+0000 mgr.smithi052.ghqrtk (mgr.14162) 328 : cluster [DBG] pgmap v312: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:20.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:20 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:20:19.191115+0000 mon.smithi052 (mon.0) 315 : audit [DBG] from='client.? 172.21.15.88:0/492326378' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:20.942 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:20.942 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:22.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:22 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:20:20.594863+0000 mgr.smithi052.ghqrtk (mgr.14162) 329 : cluster [DBG] pgmap v313: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:22.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:22.739+0000 7fec4e540700 1 -- 172.21.15.88:0/258494909 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec480f5410 msgr2=0x7fec480f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:22.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:22.739+0000 7fec4e540700 1 --2- 172.21.15.88:0/258494909 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec480f5410 0x7fec480f5830 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7fec38009fb0 tx=0x7fec38009b30).stop 2024-02-19T22:20:22.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:22.739+0000 7fec4e540700 1 -- 172.21.15.88:0/258494909 shutdown_connections 2024-02-19T22:20:22.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:22.739+0000 7fec4e540700 1 --2- 172.21.15.88:0/258494909 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec480f5410 0x7fec480f5830 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:22.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:22.739+0000 7fec4e540700 1 -- 172.21.15.88:0/258494909 >> 172.21.15.88:0/258494909 conn(0x7fec480f0b70 msgr2=0x7fec480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:22.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:22.739+0000 7fec4e540700 1 -- 172.21.15.88:0/258494909 shutdown_connections 2024-02-19T22:20:22.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:22.739+0000 7fec4e540700 1 -- 172.21.15.88:0/258494909 wait complete. 2024-02-19T22:20:22.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:22.740+0000 7fec4e540700 1 Processor -- start 2024-02-19T22:20:22.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:22.740+0000 7fec4e540700 1 -- start start 2024-02-19T22:20:22.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:22.740+0000 7fec4e540700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec480f5410 0x7fec480fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:22.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:22.740+0000 7fec4e540700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec480fb960 con 0x7fec480f5410 2024-02-19T22:20:22.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:22.740+0000 7fec47fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec480f5410 0x7fec480fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:22.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:22.740+0000 7fec47fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec480f5410 0x7fec480fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:53888/0 (socket says 172.21.15.88:53888) 2024-02-19T22:20:22.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:22.740+0000 7fec47fff700 1 -- 172.21.15.88:0/526381473 learned_addr learned my addr 172.21.15.88:0/526381473 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:20:22.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:22.741+0000 7fec47fff700 1 -- 172.21.15.88:0/526381473 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec38005040 con 0x7fec480f5410 2024-02-19T22:20:22.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:22.741+0000 7fec47fff700 1 --2- 172.21.15.88:0/526381473 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec480f5410 0x7fec480fd4e0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7fec38005010 tx=0x7fec3800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:22.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:22.741+0000 7fec44ff9700 1 -- 172.21.15.88:0/526381473 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec38009320 con 0x7fec480f5410 2024-02-19T22:20:22.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:22.741+0000 7fec4e540700 1 -- 172.21.15.88:0/526381473 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec480fbb00 con 0x7fec480f5410 2024-02-19T22:20:22.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:22.741+0000 7fec44ff9700 1 -- 172.21.15.88:0/526381473 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fec38013380 con 0x7fec480f5410 2024-02-19T22:20:22.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:22.741+0000 7fec44ff9700 1 -- 172.21.15.88:0/526381473 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec38014e50 con 0x7fec480f5410 2024-02-19T22:20:22.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:22.741+0000 7fec4e540700 1 -- 172.21.15.88:0/526381473 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec480fbf90 con 0x7fec480f5410 2024-02-19T22:20:22.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:22.742+0000 7fec4e540700 1 -- 172.21.15.88:0/526381473 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec48059100 con 0x7fec480f5410 2024-02-19T22:20:22.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:22.742+0000 7fec44ff9700 1 -- 172.21.15.88:0/526381473 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fec380095f0 con 0x7fec480f5410 2024-02-19T22:20:22.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:22.742+0000 7fec44ff9700 1 --2- 172.21.15.88:0/526381473 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fec3002ff10 0x7fec300323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:22.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:22.742+0000 7fec44ff9700 1 -- 172.21.15.88:0/526381473 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fec38040e90 con 0x7fec480f5410 2024-02-19T22:20:22.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:22.742+0000 7fec477fe700 1 --2- 172.21.15.88:0/526381473 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fec3002ff10 0x7fec300323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:22.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:22.742+0000 7fec477fe700 1 --2- 172.21.15.88:0/526381473 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fec3002ff10 0x7fec300323d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fec3c0096a0 tx=0x7fec3c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:22.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:22.746+0000 7fec44ff9700 1 -- 172.21.15.88:0/526381473 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fec38014420 con 0x7fec480f5410 2024-02-19T22:20:22.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:22.953+0000 7fec4e540700 1 -- 172.21.15.88:0/526381473 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fec48045ad0 con 0x7fec480f5410 2024-02-19T22:20:22.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:22.954+0000 7fec44ff9700 1 -- 172.21.15.88:0/526381473 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fec38010230 con 0x7fec480f5410 2024-02-19T22:20:22.956 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:20:22.956 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:22.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:22.955+0000 7fec4e540700 1 -- 172.21.15.88:0/526381473 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fec3002ff10 msgr2=0x7fec300323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:22.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:22.955+0000 7fec4e540700 1 --2- 172.21.15.88:0/526381473 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fec3002ff10 0x7fec300323d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fec3c0096a0 tx=0x7fec3c006b40).stop 2024-02-19T22:20:22.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:22.955+0000 7fec4e540700 1 -- 172.21.15.88:0/526381473 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec480f5410 msgr2=0x7fec480fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:22.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:22.955+0000 7fec4e540700 1 --2- 172.21.15.88:0/526381473 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec480f5410 0x7fec480fd4e0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7fec38005010 tx=0x7fec3800b040).stop 2024-02-19T22:20:22.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:22.955+0000 7fec4e540700 1 -- 172.21.15.88:0/526381473 shutdown_connections 2024-02-19T22:20:22.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:22.955+0000 7fec4e540700 1 --2- 172.21.15.88:0/526381473 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fec3002ff10 0x7fec300323d0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:22.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:22.955+0000 7fec4e540700 1 --2- 172.21.15.88:0/526381473 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec480f5410 0x7fec480fd4e0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:22.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:22.955+0000 7fec4e540700 1 -- 172.21.15.88:0/526381473 >> 172.21.15.88:0/526381473 conn(0x7fec480f0b70 msgr2=0x7fec480f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:22.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:22.956+0000 7fec4e540700 1 -- 172.21.15.88:0/526381473 shutdown_connections 2024-02-19T22:20:22.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:22.956+0000 7fec4e540700 1 -- 172.21.15.88:0/526381473 wait complete. 2024-02-19T22:20:22.959 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:20:23.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:23 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:20:22.955081+0000 mon.smithi052 (mon.0) 316 : audit [DBG] from='client.? 172.21.15.88:0/526381473' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:24.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:24 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:20:22.595263+0000 mgr.smithi052.ghqrtk (mgr.14162) 330 : cluster [DBG] pgmap v314: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:24.670 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:24.671 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:26.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:26.502+0000 7fa914846700 1 -- 172.21.15.88:0/1055322169 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9100f5410 msgr2=0x7fa9100f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:26.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:26.502+0000 7fa914846700 1 --2- 172.21.15.88:0/1055322169 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9100f5410 0x7fa9100f5830 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7fa8f8009fb0 tx=0x7fa8f8009b30).stop 2024-02-19T22:20:26.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:26.502+0000 7fa914846700 1 -- 172.21.15.88:0/1055322169 shutdown_connections 2024-02-19T22:20:26.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:26.502+0000 7fa914846700 1 --2- 172.21.15.88:0/1055322169 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9100f5410 0x7fa9100f5830 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:26.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:26.502+0000 7fa914846700 1 -- 172.21.15.88:0/1055322169 >> 172.21.15.88:0/1055322169 conn(0x7fa9100f0b70 msgr2=0x7fa9100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:26.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:26.503+0000 7fa914846700 1 -- 172.21.15.88:0/1055322169 shutdown_connections 2024-02-19T22:20:26.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:26.503+0000 7fa914846700 1 -- 172.21.15.88:0/1055322169 wait complete. 2024-02-19T22:20:26.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:26.503+0000 7fa914846700 1 Processor -- start 2024-02-19T22:20:26.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:26.503+0000 7fa914846700 1 -- start start 2024-02-19T22:20:26.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:26.503+0000 7fa914846700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9100f5410 0x7fa9101056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:26.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:26.503+0000 7fa914846700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa910105bd0 con 0x7fa9100f5410 2024-02-19T22:20:26.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:26.504+0000 7fa90e59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9100f5410 0x7fa9101056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:26.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:26.504+0000 7fa90e59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9100f5410 0x7fa9101056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:52334/0 (socket says 172.21.15.88:52334) 2024-02-19T22:20:26.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:26.504+0000 7fa90e59c700 1 -- 172.21.15.88:0/2551255643 learned_addr learned my addr 172.21.15.88:0/2551255643 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:20:26.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:26.504+0000 7fa90e59c700 1 -- 172.21.15.88:0/2551255643 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f8005040 con 0x7fa9100f5410 2024-02-19T22:20:26.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:26.504+0000 7fa90e59c700 1 --2- 172.21.15.88:0/2551255643 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9100f5410 0x7fa9101056e0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7fa8f80089f0 tx=0x7fa8f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:26.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:26.504+0000 7fa906ffd700 1 -- 172.21.15.88:0/2551255643 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8f8009320 con 0x7fa9100f5410 2024-02-19T22:20:26.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:26.504+0000 7fa914846700 1 -- 172.21.15.88:0/2551255643 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa910105dd0 con 0x7fa9100f5410 2024-02-19T22:20:26.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:26.504+0000 7fa906ffd700 1 -- 172.21.15.88:0/2551255643 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa8f8013380 con 0x7fa9100f5410 2024-02-19T22:20:26.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:26.504+0000 7fa906ffd700 1 -- 172.21.15.88:0/2551255643 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8f801ed80 con 0x7fa9100f5410 2024-02-19T22:20:26.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:26.504+0000 7fa914846700 1 -- 172.21.15.88:0/2551255643 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa910106260 con 0x7fa9100f5410 2024-02-19T22:20:26.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:26.505+0000 7fa906ffd700 1 -- 172.21.15.88:0/2551255643 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa8f801e420 con 0x7fa9100f5410 2024-02-19T22:20:26.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:26.505+0000 7fa914846700 1 -- 172.21.15.88:0/2551255643 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa910045ad0 con 0x7fa9100f5410 2024-02-19T22:20:26.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:26.505+0000 7fa906ffd700 1 --2- 172.21.15.88:0/2551255643 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa8fc02ff60 0x7fa8fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:26.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:26.505+0000 7fa906ffd700 1 -- 172.21.15.88:0/2551255643 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa8f8041150 con 0x7fa9100f5410 2024-02-19T22:20:26.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:26.505+0000 7fa90dd9b700 1 --2- 172.21.15.88:0/2551255643 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa8fc02ff60 0x7fa8fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:26.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:26.506+0000 7fa90dd9b700 1 --2- 172.21.15.88:0/2551255643 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa8fc02ff60 0x7fa8fc032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fa9000096a0 tx=0x7fa900006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:26.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:26.510+0000 7fa906ffd700 1 -- 172.21.15.88:0/2551255643 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa8f80095f0 con 0x7fa9100f5410 2024-02-19T22:20:26.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:26 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:20:24.595530+0000 mgr.smithi052.ghqrtk (mgr.14162) 331 : cluster [DBG] pgmap v315: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:26.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:26.714+0000 7fa914846700 1 -- 172.21.15.88:0/2551255643 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa910107110 con 0x7fa9100f5410 2024-02-19T22:20:26.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:26.715+0000 7fa906ffd700 1 -- 172.21.15.88:0/2551255643 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa8f8004490 con 0x7fa9100f5410 2024-02-19T22:20:26.718 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:20:26.718 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:26.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:26.716+0000 7fa914846700 1 -- 172.21.15.88:0/2551255643 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa8fc02ff60 msgr2=0x7fa8fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:26.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:26.716+0000 7fa914846700 1 --2- 172.21.15.88:0/2551255643 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa8fc02ff60 0x7fa8fc032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fa9000096a0 tx=0x7fa900006b40).stop 2024-02-19T22:20:26.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:26.716+0000 7fa914846700 1 -- 172.21.15.88:0/2551255643 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9100f5410 msgr2=0x7fa9101056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:26.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:26.716+0000 7fa914846700 1 --2- 172.21.15.88:0/2551255643 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9100f5410 0x7fa9101056e0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7fa8f80089f0 tx=0x7fa8f800b040).stop 2024-02-19T22:20:26.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:26.716+0000 7fa914846700 1 -- 172.21.15.88:0/2551255643 shutdown_connections 2024-02-19T22:20:26.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:26.716+0000 7fa914846700 1 --2- 172.21.15.88:0/2551255643 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa8fc02ff60 0x7fa8fc032420 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:26.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:26.716+0000 7fa914846700 1 --2- 172.21.15.88:0/2551255643 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9100f5410 0x7fa9101056e0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:26.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:26.716+0000 7fa914846700 1 -- 172.21.15.88:0/2551255643 >> 172.21.15.88:0/2551255643 conn(0x7fa9100f0b70 msgr2=0x7fa9100f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:26.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:26.717+0000 7fa914846700 1 -- 172.21.15.88:0/2551255643 shutdown_connections 2024-02-19T22:20:26.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:26.717+0000 7fa914846700 1 -- 172.21.15.88:0/2551255643 wait complete. 2024-02-19T22:20:26.720 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:20:27.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:27 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:20:26.716276+0000 mon.smithi052 (mon.0) 317 : audit [DBG] from='client.? 172.21.15.88:0/2551255643' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:28.387 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:28.388 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:28.594 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:28 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:20:26.595848+0000 mgr.smithi052.ghqrtk (mgr.14162) 332 : cluster [DBG] pgmap v316: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:30.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:30.210+0000 7f3d48370700 1 -- 172.21.15.88:0/2999606180 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3d400f6510 msgr2=0x7f3d400f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:30.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:30.210+0000 7f3d48370700 1 --2- 172.21.15.88:0/2999606180 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3d400f6510 0x7f3d400f6930 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f3d34009fb0 tx=0x7f3d34009b30).stop 2024-02-19T22:20:30.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:30.211+0000 7f3d48370700 1 -- 172.21.15.88:0/2999606180 shutdown_connections 2024-02-19T22:20:30.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:30.211+0000 7f3d48370700 1 --2- 172.21.15.88:0/2999606180 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3d400f6510 0x7f3d400f6930 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:30.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:30.211+0000 7f3d48370700 1 -- 172.21.15.88:0/2999606180 >> 172.21.15.88:0/2999606180 conn(0x7f3d400f1c70 msgr2=0x7f3d400f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:30.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:30.211+0000 7f3d48370700 1 -- 172.21.15.88:0/2999606180 shutdown_connections 2024-02-19T22:20:30.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:30.211+0000 7f3d48370700 1 -- 172.21.15.88:0/2999606180 wait complete. 2024-02-19T22:20:30.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:30.211+0000 7f3d48370700 1 Processor -- start 2024-02-19T22:20:30.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:30.211+0000 7f3d48370700 1 -- start start 2024-02-19T22:20:30.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:30.211+0000 7f3d48370700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3d400f6510 0x7f3d401067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:30.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:30.211+0000 7f3d48370700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d40106cd0 con 0x7f3d400f6510 2024-02-19T22:20:30.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:30.211+0000 7f3d4610c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3d400f6510 0x7f3d401067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:30.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:30.211+0000 7f3d4610c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3d400f6510 0x7f3d401067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:52354/0 (socket says 172.21.15.88:52354) 2024-02-19T22:20:30.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:30.211+0000 7f3d4610c700 1 -- 172.21.15.88:0/3357549682 learned_addr learned my addr 172.21.15.88:0/3357549682 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:20:30.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:30.212+0000 7f3d4610c700 1 -- 172.21.15.88:0/3357549682 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d34005040 con 0x7f3d400f6510 2024-02-19T22:20:30.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:30.212+0000 7f3d4610c700 1 --2- 172.21.15.88:0/3357549682 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3d400f6510 0x7f3d401067e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f3d340089f0 tx=0x7f3d3400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:30.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:30.212+0000 7f3d32ffd700 1 -- 172.21.15.88:0/3357549682 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d34009320 con 0x7f3d400f6510 2024-02-19T22:20:30.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:30.212+0000 7f3d48370700 1 -- 172.21.15.88:0/3357549682 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d40106ed0 con 0x7f3d400f6510 2024-02-19T22:20:30.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:30.212+0000 7f3d32ffd700 1 -- 172.21.15.88:0/3357549682 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3d34013380 con 0x7f3d400f6510 2024-02-19T22:20:30.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:30.212+0000 7f3d32ffd700 1 -- 172.21.15.88:0/3357549682 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d3401ed80 con 0x7f3d400f6510 2024-02-19T22:20:30.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:30.212+0000 7f3d48370700 1 -- 172.21.15.88:0/3357549682 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d40107360 con 0x7f3d400f6510 2024-02-19T22:20:30.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:30.213+0000 7f3d48370700 1 -- 172.21.15.88:0/3357549682 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d40045ad0 con 0x7f3d400f6510 2024-02-19T22:20:30.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:30.213+0000 7f3d32ffd700 1 -- 172.21.15.88:0/3357549682 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3d3401e420 con 0x7f3d400f6510 2024-02-19T22:20:30.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:30.213+0000 7f3d32ffd700 1 --2- 172.21.15.88:0/3357549682 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f3d2c02ff60 0x7f3d2c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:30.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:30.213+0000 7f3d32ffd700 1 -- 172.21.15.88:0/3357549682 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3d340410b0 con 0x7f3d400f6510 2024-02-19T22:20:30.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:30.213+0000 7f3d4590b700 1 --2- 172.21.15.88:0/3357549682 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f3d2c02ff60 0x7f3d2c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:30.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:30.213+0000 7f3d4590b700 1 --2- 172.21.15.88:0/3357549682 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f3d2c02ff60 0x7f3d2c032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f3d3c0096a0 tx=0x7f3d3c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:30.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:30.218+0000 7f3d32ffd700 1 -- 172.21.15.88:0/3357549682 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3d340095f0 con 0x7f3d400f6510 2024-02-19T22:20:30.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:30.422+0000 7f3d48370700 1 -- 172.21.15.88:0/3357549682 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3d401081b0 con 0x7f3d400f6510 2024-02-19T22:20:30.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:30.423+0000 7f3d32ffd700 1 -- 172.21.15.88:0/3357549682 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3d3401cb10 con 0x7f3d400f6510 2024-02-19T22:20:30.426 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:20:30.426 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:30.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:30.425+0000 7f3d48370700 1 -- 172.21.15.88:0/3357549682 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f3d2c02ff60 msgr2=0x7f3d2c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:30.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:30.425+0000 7f3d48370700 1 --2- 172.21.15.88:0/3357549682 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f3d2c02ff60 0x7f3d2c032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f3d3c0096a0 tx=0x7f3d3c006b40).stop 2024-02-19T22:20:30.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:30.425+0000 7f3d48370700 1 -- 172.21.15.88:0/3357549682 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3d400f6510 msgr2=0x7f3d401067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:30.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:30.425+0000 7f3d48370700 1 --2- 172.21.15.88:0/3357549682 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3d400f6510 0x7f3d401067e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f3d340089f0 tx=0x7f3d3400b040).stop 2024-02-19T22:20:30.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:30.425+0000 7f3d48370700 1 -- 172.21.15.88:0/3357549682 shutdown_connections 2024-02-19T22:20:30.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:30.425+0000 7f3d48370700 1 --2- 172.21.15.88:0/3357549682 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f3d2c02ff60 0x7f3d2c032420 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:30.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:30.425+0000 7f3d48370700 1 --2- 172.21.15.88:0/3357549682 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3d400f6510 0x7f3d401067e0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:30.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:30.425+0000 7f3d48370700 1 -- 172.21.15.88:0/3357549682 >> 172.21.15.88:0/3357549682 conn(0x7f3d400f1c70 msgr2=0x7f3d400f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:30.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:30.425+0000 7f3d48370700 1 -- 172.21.15.88:0/3357549682 shutdown_connections 2024-02-19T22:20:30.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:30.425+0000 7f3d48370700 1 -- 172.21.15.88:0/3357549682 wait complete. 2024-02-19T22:20:30.429 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:20:30.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:30 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:20:28.596344+0000 mgr.smithi052.ghqrtk (mgr.14162) 333 : cluster [DBG] pgmap v317: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:31.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:31 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:20:30.424559+0000 mon.smithi052 (mon.0) 318 : audit [DBG] from='client.? 172.21.15.88:0/3357549682' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:32.032 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:32.033 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:32.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:32 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:20:30.596659+0000 mgr.smithi052.ghqrtk (mgr.14162) 334 : cluster [DBG] pgmap v318: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:33.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:33.941+0000 7f020410d700 1 -- 172.21.15.88:0/1058876853 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f01fc0f6510 msgr2=0x7f01fc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:33.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:33.941+0000 7f020410d700 1 --2- 172.21.15.88:0/1058876853 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f01fc0f6510 0x7f01fc0f6930 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f01ec009fb0 tx=0x7f01ec009b30).stop 2024-02-19T22:20:33.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:33.941+0000 7f020410d700 1 -- 172.21.15.88:0/1058876853 shutdown_connections 2024-02-19T22:20:33.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:33.941+0000 7f020410d700 1 --2- 172.21.15.88:0/1058876853 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f01fc0f6510 0x7f01fc0f6930 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:33.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:33.941+0000 7f020410d700 1 -- 172.21.15.88:0/1058876853 >> 172.21.15.88:0/1058876853 conn(0x7f01fc0f1c70 msgr2=0x7f01fc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:33.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:33.941+0000 7f020410d700 1 -- 172.21.15.88:0/1058876853 shutdown_connections 2024-02-19T22:20:33.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:33.941+0000 7f020410d700 1 -- 172.21.15.88:0/1058876853 wait complete. 2024-02-19T22:20:33.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:33.942+0000 7f020410d700 1 Processor -- start 2024-02-19T22:20:33.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:33.942+0000 7f020410d700 1 -- start start 2024-02-19T22:20:33.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:33.942+0000 7f020410d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f01fc0f6510 0x7f01fc1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:33.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:33.942+0000 7f020410d700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01fc106cd0 con 0x7f01fc0f6510 2024-02-19T22:20:33.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:33.942+0000 7f0201ea9700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f01fc0f6510 0x7f01fc1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:33.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:33.942+0000 7f0201ea9700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f01fc0f6510 0x7f01fc1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:33032/0 (socket says 172.21.15.88:33032) 2024-02-19T22:20:33.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:33.942+0000 7f0201ea9700 1 -- 172.21.15.88:0/1976377562 learned_addr learned my addr 172.21.15.88:0/1976377562 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:20:33.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:33.943+0000 7f0201ea9700 1 -- 172.21.15.88:0/1976377562 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01ec005040 con 0x7f01fc0f6510 2024-02-19T22:20:33.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:33.943+0000 7f0201ea9700 1 --2- 172.21.15.88:0/1976377562 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f01fc0f6510 0x7f01fc1067e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f01ec005010 tx=0x7f01ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:33.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:33.943+0000 7f01f27fc700 1 -- 172.21.15.88:0/1976377562 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01ec009320 con 0x7f01fc0f6510 2024-02-19T22:20:33.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:33.943+0000 7f020410d700 1 -- 172.21.15.88:0/1976377562 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01fc106ed0 con 0x7f01fc0f6510 2024-02-19T22:20:33.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:33.943+0000 7f01f27fc700 1 -- 172.21.15.88:0/1976377562 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f01ec013380 con 0x7f01fc0f6510 2024-02-19T22:20:33.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:33.943+0000 7f01f27fc700 1 -- 172.21.15.88:0/1976377562 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01ec014e50 con 0x7f01fc0f6510 2024-02-19T22:20:33.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:33.943+0000 7f020410d700 1 -- 172.21.15.88:0/1976377562 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01fc107360 con 0x7f01fc0f6510 2024-02-19T22:20:33.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:33.944+0000 7f01f27fc700 1 -- 172.21.15.88:0/1976377562 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f01ec0095f0 con 0x7f01fc0f6510 2024-02-19T22:20:33.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:33.944+0000 7f020410d700 1 -- 172.21.15.88:0/1976377562 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01fc045ad0 con 0x7f01fc0f6510 2024-02-19T22:20:33.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:33.944+0000 7f01f27fc700 1 --2- 172.21.15.88:0/1976377562 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f01e802ff10 0x7f01e80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:33.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:33.944+0000 7f01f27fc700 1 -- 172.21.15.88:0/1976377562 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f01ec040e90 con 0x7f01fc0f6510 2024-02-19T22:20:33.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:33.944+0000 7f02016a8700 1 --2- 172.21.15.88:0/1976377562 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f01e802ff10 0x7f01e80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:33.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:33.944+0000 7f02016a8700 1 --2- 172.21.15.88:0/1976377562 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f01e802ff10 0x7f01e80323d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f01f80096a0 tx=0x7f01f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:33.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:33.949+0000 7f01f27fc700 1 -- 172.21.15.88:0/1976377562 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f01ec014420 con 0x7f01fc0f6510 2024-02-19T22:20:34.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:34.152+0000 7f020410d700 1 -- 172.21.15.88:0/1976377562 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f01fc108210 con 0x7f01fc0f6510 2024-02-19T22:20:34.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:34.152+0000 7f01f27fc700 1 -- 172.21.15.88:0/1976377562 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f01ec010230 con 0x7f01fc0f6510 2024-02-19T22:20:34.155 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:20:34.155 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:34.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:34.154+0000 7f020410d700 1 -- 172.21.15.88:0/1976377562 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f01e802ff10 msgr2=0x7f01e80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:34.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:34.154+0000 7f020410d700 1 --2- 172.21.15.88:0/1976377562 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f01e802ff10 0x7f01e80323d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f01f80096a0 tx=0x7f01f8006b40).stop 2024-02-19T22:20:34.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:34.154+0000 7f020410d700 1 -- 172.21.15.88:0/1976377562 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f01fc0f6510 msgr2=0x7f01fc1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:34.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:34.154+0000 7f020410d700 1 --2- 172.21.15.88:0/1976377562 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f01fc0f6510 0x7f01fc1067e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f01ec005010 tx=0x7f01ec00b040).stop 2024-02-19T22:20:34.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:34.154+0000 7f020410d700 1 -- 172.21.15.88:0/1976377562 shutdown_connections 2024-02-19T22:20:34.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:34.154+0000 7f020410d700 1 --2- 172.21.15.88:0/1976377562 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f01e802ff10 0x7f01e80323d0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:34.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:34.154+0000 7f020410d700 1 --2- 172.21.15.88:0/1976377562 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f01fc0f6510 0x7f01fc1067e0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:34.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:34.154+0000 7f020410d700 1 -- 172.21.15.88:0/1976377562 >> 172.21.15.88:0/1976377562 conn(0x7f01fc0f1c70 msgr2=0x7f01fc0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:34.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:34.154+0000 7f020410d700 1 -- 172.21.15.88:0/1976377562 shutdown_connections 2024-02-19T22:20:34.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:34.154+0000 7f020410d700 1 -- 172.21.15.88:0/1976377562 wait complete. 2024-02-19T22:20:34.157 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:20:34.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:34 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:20:32.596926+0000 mgr.smithi052.ghqrtk (mgr.14162) 335 : cluster [DBG] pgmap v319: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:34.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:34 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:20:34.153752+0000 mon.smithi052 (mon.0) 319 : audit [DBG] from='client.? 172.21.15.88:0/1976377562' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:35.739 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:35.739 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:36.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:36 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:20:34.597338+0000 mgr.smithi052.ghqrtk (mgr.14162) 336 : cluster [DBG] pgmap v320: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:37.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:37.631+0000 7fb1d578e700 1 -- 172.21.15.88:0/2046030673 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb1d00f51a0 msgr2=0x7fb1d00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:37.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:37.631+0000 7fb1d578e700 1 --2- 172.21.15.88:0/2046030673 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb1d00f51a0 0x7fb1d00f55c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7fb1c0009fb0 tx=0x7fb1c0009b30).stop 2024-02-19T22:20:37.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:37.631+0000 7fb1d578e700 1 -- 172.21.15.88:0/2046030673 shutdown_connections 2024-02-19T22:20:37.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:37.631+0000 7fb1d578e700 1 --2- 172.21.15.88:0/2046030673 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb1d00f51a0 0x7fb1d00f55c0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:37.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:37.631+0000 7fb1d578e700 1 -- 172.21.15.88:0/2046030673 >> 172.21.15.88:0/2046030673 conn(0x7fb1d00f09a0 msgr2=0x7fb1d00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:37.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:37.631+0000 7fb1d578e700 1 -- 172.21.15.88:0/2046030673 shutdown_connections 2024-02-19T22:20:37.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:37.631+0000 7fb1d578e700 1 -- 172.21.15.88:0/2046030673 wait complete. 2024-02-19T22:20:37.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:37.631+0000 7fb1d578e700 1 Processor -- start 2024-02-19T22:20:37.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:37.631+0000 7fb1d578e700 1 -- start start 2024-02-19T22:20:37.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:37.631+0000 7fb1d578e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb1d00f51a0 0x7fb1d0109290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:37.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:37.631+0000 7fb1d578e700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d0060280 con 0x7fb1d00f51a0 2024-02-19T22:20:37.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:37.632+0000 7fb1cffff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb1d00f51a0 0x7fb1d0109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:37.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:37.632+0000 7fb1cffff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb1d00f51a0 0x7fb1d0109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:33038/0 (socket says 172.21.15.88:33038) 2024-02-19T22:20:37.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:37.632+0000 7fb1cffff700 1 -- 172.21.15.88:0/1248605253 learned_addr learned my addr 172.21.15.88:0/1248605253 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:20:37.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:37.632+0000 7fb1cffff700 1 -- 172.21.15.88:0/1248605253 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1c0005040 con 0x7fb1d00f51a0 2024-02-19T22:20:37.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:37.632+0000 7fb1cffff700 1 --2- 172.21.15.88:0/1248605253 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb1d00f51a0 0x7fb1d0109290 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7fb1c00089f0 tx=0x7fb1c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:37.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:37.632+0000 7fb1ccff9700 1 -- 172.21.15.88:0/1248605253 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1c0009320 con 0x7fb1d00f51a0 2024-02-19T22:20:37.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:37.632+0000 7fb1d578e700 1 -- 172.21.15.88:0/1248605253 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1d0060480 con 0x7fb1d00f51a0 2024-02-19T22:20:37.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:37.632+0000 7fb1ccff9700 1 -- 172.21.15.88:0/1248605253 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb1c0013380 con 0x7fb1d00f51a0 2024-02-19T22:20:37.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:37.632+0000 7fb1ccff9700 1 -- 172.21.15.88:0/1248605253 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1c0014e20 con 0x7fb1d00f51a0 2024-02-19T22:20:37.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:37.632+0000 7fb1d578e700 1 -- 172.21.15.88:0/1248605253 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1d01099c0 con 0x7fb1d00f51a0 2024-02-19T22:20:37.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:37.633+0000 7fb1ccff9700 1 -- 172.21.15.88:0/1248605253 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb1c00095f0 con 0x7fb1d00f51a0 2024-02-19T22:20:37.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:37.633+0000 7fb1d578e700 1 -- 172.21.15.88:0/1248605253 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1d0045ad0 con 0x7fb1d00f51a0 2024-02-19T22:20:37.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:37.633+0000 7fb1ccff9700 1 --2- 172.21.15.88:0/1248605253 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fb1b802ff10 0x7fb1b80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:37.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:37.633+0000 7fb1ccff9700 1 -- 172.21.15.88:0/1248605253 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb1c0040e90 con 0x7fb1d00f51a0 2024-02-19T22:20:37.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:37.634+0000 7fb1cf7fe700 1 --2- 172.21.15.88:0/1248605253 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fb1b802ff10 0x7fb1b80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:37.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:37.634+0000 7fb1cf7fe700 1 --2- 172.21.15.88:0/1248605253 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fb1b802ff10 0x7fb1b80323d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fb1c40096a0 tx=0x7fb1c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:37.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:37.638+0000 7fb1ccff9700 1 -- 172.21.15.88:0/1248605253 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb1c0014420 con 0x7fb1d00f51a0 2024-02-19T22:20:37.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:37.841+0000 7fb1d578e700 1 -- 172.21.15.88:0/1248605253 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb1d00fad30 con 0x7fb1d00f51a0 2024-02-19T22:20:37.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:37.842+0000 7fb1ccff9700 1 -- 172.21.15.88:0/1248605253 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb1c001d980 con 0x7fb1d00f51a0 2024-02-19T22:20:37.845 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:20:37.845 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:37.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:37.843+0000 7fb1d578e700 1 -- 172.21.15.88:0/1248605253 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fb1b802ff10 msgr2=0x7fb1b80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:37.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:37.843+0000 7fb1d578e700 1 --2- 172.21.15.88:0/1248605253 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fb1b802ff10 0x7fb1b80323d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fb1c40096a0 tx=0x7fb1c4006b40).stop 2024-02-19T22:20:37.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:37.843+0000 7fb1d578e700 1 -- 172.21.15.88:0/1248605253 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb1d00f51a0 msgr2=0x7fb1d0109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:37.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:37.843+0000 7fb1d578e700 1 --2- 172.21.15.88:0/1248605253 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb1d00f51a0 0x7fb1d0109290 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7fb1c00089f0 tx=0x7fb1c000b040).stop 2024-02-19T22:20:37.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:37.843+0000 7fb1d578e700 1 -- 172.21.15.88:0/1248605253 shutdown_connections 2024-02-19T22:20:37.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:37.843+0000 7fb1d578e700 1 --2- 172.21.15.88:0/1248605253 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fb1b802ff10 0x7fb1b80323d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:37.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:37.843+0000 7fb1d578e700 1 --2- 172.21.15.88:0/1248605253 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb1d00f51a0 0x7fb1d0109290 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:37.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:37.843+0000 7fb1d578e700 1 -- 172.21.15.88:0/1248605253 >> 172.21.15.88:0/1248605253 conn(0x7fb1d00f09a0 msgr2=0x7fb1d00f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:37.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:37.844+0000 7fb1d578e700 1 -- 172.21.15.88:0/1248605253 shutdown_connections 2024-02-19T22:20:37.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:37.844+0000 7fb1d578e700 1 -- 172.21.15.88:0/1248605253 wait complete. 2024-02-19T22:20:37.847 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:20:38.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:38 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:20:36.597605+0000 mgr.smithi052.ghqrtk (mgr.14162) 337 : cluster [DBG] pgmap v321: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:38.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:38 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:20:37.843276+0000 mon.smithi052 (mon.0) 320 : audit [DBG] from='client.? 172.21.15.88:0/1248605253' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:39.673 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:39.674 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:40.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:40 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:20:38.597990+0000 mgr.smithi052.ghqrtk (mgr.14162) 338 : cluster [DBG] pgmap v322: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:41.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:41.501+0000 7f0f624b8700 1 -- 172.21.15.88:0/3314133266 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f5c0f7b60 msgr2=0x7f0f5c0f7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:41.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:41.501+0000 7f0f624b8700 1 --2- 172.21.15.88:0/3314133266 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f5c0f7b60 0x7f0f5c0f7f80 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f0f50009fb0 tx=0x7f0f50009b30).stop 2024-02-19T22:20:41.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:41.501+0000 7f0f624b8700 1 -- 172.21.15.88:0/3314133266 shutdown_connections 2024-02-19T22:20:41.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:41.501+0000 7f0f624b8700 1 --2- 172.21.15.88:0/3314133266 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f5c0f7b60 0x7f0f5c0f7f80 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:41.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:41.501+0000 7f0f624b8700 1 -- 172.21.15.88:0/3314133266 >> 172.21.15.88:0/3314133266 conn(0x7f0f5c0f02e0 msgr2=0x7f0f5c0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:41.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:41.501+0000 7f0f624b8700 1 -- 172.21.15.88:0/3314133266 shutdown_connections 2024-02-19T22:20:41.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:41.501+0000 7f0f624b8700 1 -- 172.21.15.88:0/3314133266 wait complete. 2024-02-19T22:20:41.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:41.501+0000 7f0f624b8700 1 Processor -- start 2024-02-19T22:20:41.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:41.502+0000 7f0f624b8700 1 -- start start 2024-02-19T22:20:41.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:41.502+0000 7f0f624b8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f5c0f7b60 0x7f0f5c106830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:41.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:41.502+0000 7f0f624b8700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5c106d20 con 0x7f0f5c0f7b60 2024-02-19T22:20:41.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:41.502+0000 7f0f5bfff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f5c0f7b60 0x7f0f5c106830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:41.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:41.502+0000 7f0f5bfff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f5c0f7b60 0x7f0f5c106830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:33050/0 (socket says 172.21.15.88:33050) 2024-02-19T22:20:41.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:41.502+0000 7f0f5bfff700 1 -- 172.21.15.88:0/399311480 learned_addr learned my addr 172.21.15.88:0/399311480 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:20:41.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:41.502+0000 7f0f5bfff700 1 -- 172.21.15.88:0/399311480 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f50005040 con 0x7f0f5c0f7b60 2024-02-19T22:20:41.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:41.503+0000 7f0f5bfff700 1 --2- 172.21.15.88:0/399311480 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f5c0f7b60 0x7f0f5c106830 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f0f50005010 tx=0x7f0f5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:41.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:41.503+0000 7f0f58ff9700 1 -- 172.21.15.88:0/399311480 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f50009320 con 0x7f0f5c0f7b60 2024-02-19T22:20:41.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:41.503+0000 7f0f624b8700 1 -- 172.21.15.88:0/399311480 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f5c106f20 con 0x7f0f5c0f7b60 2024-02-19T22:20:41.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:41.503+0000 7f0f58ff9700 1 -- 172.21.15.88:0/399311480 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0f50013380 con 0x7f0f5c0f7b60 2024-02-19T22:20:41.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:41.503+0000 7f0f58ff9700 1 -- 172.21.15.88:0/399311480 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f50014e50 con 0x7f0f5c0f7b60 2024-02-19T22:20:41.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:41.503+0000 7f0f624b8700 1 -- 172.21.15.88:0/399311480 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f5c1073b0 con 0x7f0f5c0f7b60 2024-02-19T22:20:41.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:41.504+0000 7f0f58ff9700 1 -- 172.21.15.88:0/399311480 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0f500095f0 con 0x7f0f5c0f7b60 2024-02-19T22:20:41.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:41.504+0000 7f0f624b8700 1 -- 172.21.15.88:0/399311480 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f5c045ad0 con 0x7f0f5c0f7b60 2024-02-19T22:20:41.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:41.504+0000 7f0f58ff9700 1 --2- 172.21.15.88:0/399311480 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f0f4402ff10 0x7f0f440323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:41.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:41.504+0000 7f0f58ff9700 1 -- 172.21.15.88:0/399311480 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0f50040fa0 con 0x7f0f5c0f7b60 2024-02-19T22:20:41.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:41.504+0000 7f0f5b7fe700 1 --2- 172.21.15.88:0/399311480 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f0f4402ff10 0x7f0f440323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:41.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:41.504+0000 7f0f5b7fe700 1 --2- 172.21.15.88:0/399311480 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f0f4402ff10 0x7f0f440323d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f0f4c0096a0 tx=0x7f0f4c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:41.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:41.509+0000 7f0f58ff9700 1 -- 172.21.15.88:0/399311480 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0f50014420 con 0x7f0f5c0f7b60 2024-02-19T22:20:41.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:41.714+0000 7f0f624b8700 1 -- 172.21.15.88:0/399311480 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0f5c108260 con 0x7f0f5c0f7b60 2024-02-19T22:20:41.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:41.715+0000 7f0f58ff9700 1 -- 172.21.15.88:0/399311480 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0f50010230 con 0x7f0f5c0f7b60 2024-02-19T22:20:41.717 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:20:41.717 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:41.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:41.716+0000 7f0f624b8700 1 -- 172.21.15.88:0/399311480 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f0f4402ff10 msgr2=0x7f0f440323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:41.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:41.716+0000 7f0f624b8700 1 --2- 172.21.15.88:0/399311480 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f0f4402ff10 0x7f0f440323d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f0f4c0096a0 tx=0x7f0f4c006b40).stop 2024-02-19T22:20:41.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:41.716+0000 7f0f624b8700 1 -- 172.21.15.88:0/399311480 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f5c0f7b60 msgr2=0x7f0f5c106830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:41.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:41.716+0000 7f0f624b8700 1 --2- 172.21.15.88:0/399311480 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f5c0f7b60 0x7f0f5c106830 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f0f50005010 tx=0x7f0f5000b040).stop 2024-02-19T22:20:41.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:41.716+0000 7f0f624b8700 1 -- 172.21.15.88:0/399311480 shutdown_connections 2024-02-19T22:20:41.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:41.716+0000 7f0f624b8700 1 --2- 172.21.15.88:0/399311480 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f0f4402ff10 0x7f0f440323d0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:41.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:41.716+0000 7f0f624b8700 1 --2- 172.21.15.88:0/399311480 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f5c0f7b60 0x7f0f5c106830 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:41.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:41.716+0000 7f0f624b8700 1 -- 172.21.15.88:0/399311480 >> 172.21.15.88:0/399311480 conn(0x7f0f5c0f02e0 msgr2=0x7f0f5c0f0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:41.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:41.716+0000 7f0f624b8700 1 -- 172.21.15.88:0/399311480 shutdown_connections 2024-02-19T22:20:41.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:41.716+0000 7f0f624b8700 1 -- 172.21.15.88:0/399311480 wait complete. 2024-02-19T22:20:41.720 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:20:42.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:42 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:20:40.598387+0000 mgr.smithi052.ghqrtk (mgr.14162) 339 : cluster [DBG] pgmap v323: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:42.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:42 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:20:41.716158+0000 mon.smithi052 (mon.0) 321 : audit [DBG] from='client.? 172.21.15.88:0/399311480' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:43.474 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:43.474 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:44.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:44 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:20:42.598727+0000 mgr.smithi052.ghqrtk (mgr.14162) 340 : cluster [DBG] pgmap v324: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:44.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:44 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:20:43.756288+0000 mon.smithi052 (mon.0) 322 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi052.ghqrtk/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:20:44.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:44 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:20:43.758331+0000 mon.smithi052 (mon.0) 323 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi052.ghqrtk/trash_purge_schedule"}]: dispatch 2024-02-19T22:20:45.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:45.375+0000 7f4a74183700 1 -- 172.21.15.88:0/1000098084 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4a6c0f5410 msgr2=0x7f4a6c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:45.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:45.375+0000 7f4a74183700 1 --2- 172.21.15.88:0/1000098084 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4a6c0f5410 0x7f4a6c0f5830 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f4a5c009fb0 tx=0x7f4a5c009b30).stop 2024-02-19T22:20:45.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:45.375+0000 7f4a74183700 1 -- 172.21.15.88:0/1000098084 shutdown_connections 2024-02-19T22:20:45.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:45.375+0000 7f4a74183700 1 --2- 172.21.15.88:0/1000098084 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4a6c0f5410 0x7f4a6c0f5830 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:45.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:45.375+0000 7f4a74183700 1 -- 172.21.15.88:0/1000098084 >> 172.21.15.88:0/1000098084 conn(0x7f4a6c0f0b70 msgr2=0x7f4a6c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:45.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:45.375+0000 7f4a74183700 1 -- 172.21.15.88:0/1000098084 shutdown_connections 2024-02-19T22:20:45.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:45.375+0000 7f4a74183700 1 -- 172.21.15.88:0/1000098084 wait complete. 2024-02-19T22:20:45.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:45.376+0000 7f4a74183700 1 Processor -- start 2024-02-19T22:20:45.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:45.376+0000 7f4a74183700 1 -- start start 2024-02-19T22:20:45.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:45.376+0000 7f4a74183700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4a6c0f5410 0x7f4a6c0fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:45.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:45.376+0000 7f4a74183700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6c0fb960 con 0x7f4a6c0f5410 2024-02-19T22:20:45.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:45.376+0000 7f4a71f1f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4a6c0f5410 0x7f4a6c0fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:45.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:45.376+0000 7f4a71f1f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4a6c0f5410 0x7f4a6c0fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:58680/0 (socket says 172.21.15.88:58680) 2024-02-19T22:20:45.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:45.376+0000 7f4a71f1f700 1 -- 172.21.15.88:0/3550597007 learned_addr learned my addr 172.21.15.88:0/3550597007 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:20:45.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:45.376+0000 7f4a71f1f700 1 -- 172.21.15.88:0/3550597007 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a5c005040 con 0x7f4a6c0f5410 2024-02-19T22:20:45.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:45.377+0000 7f4a71f1f700 1 --2- 172.21.15.88:0/3550597007 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4a6c0f5410 0x7f4a6c0fd4e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f4a5c005010 tx=0x7f4a5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:45.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:45.377+0000 7f4a627fc700 1 -- 172.21.15.88:0/3550597007 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a5c009320 con 0x7f4a6c0f5410 2024-02-19T22:20:45.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:45.377+0000 7f4a74183700 1 -- 172.21.15.88:0/3550597007 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a6c0fbb00 con 0x7f4a6c0f5410 2024-02-19T22:20:45.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:45.377+0000 7f4a627fc700 1 -- 172.21.15.88:0/3550597007 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4a5c013380 con 0x7f4a6c0f5410 2024-02-19T22:20:45.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:45.377+0000 7f4a627fc700 1 -- 172.21.15.88:0/3550597007 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a5c014e50 con 0x7f4a6c0f5410 2024-02-19T22:20:45.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:45.377+0000 7f4a74183700 1 -- 172.21.15.88:0/3550597007 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a6c0fbf90 con 0x7f4a6c0f5410 2024-02-19T22:20:45.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:45.378+0000 7f4a627fc700 1 -- 172.21.15.88:0/3550597007 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4a5c0263a0 con 0x7f4a6c0f5410 2024-02-19T22:20:45.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:45.378+0000 7f4a627fc700 1 --2- 172.21.15.88:0/3550597007 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f4a5802ff10 0x7f4a580323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:45.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:45.378+0000 7f4a627fc700 1 -- 172.21.15.88:0/3550597007 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4a5c040e60 con 0x7f4a6c0f5410 2024-02-19T22:20:45.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:45.378+0000 7f4a74183700 1 -- 172.21.15.88:0/3550597007 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a6c059100 con 0x7f4a6c0f5410 2024-02-19T22:20:45.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:45.378+0000 7f4a7171e700 1 --2- 172.21.15.88:0/3550597007 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f4a5802ff10 0x7f4a580323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:45.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:45.379+0000 7f4a7171e700 1 --2- 172.21.15.88:0/3550597007 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f4a5802ff10 0x7f4a580323d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f4a680096a0 tx=0x7f4a68006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:45.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:45.383+0000 7f4a627fc700 1 -- 172.21.15.88:0/3550597007 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4a5c012920 con 0x7f4a6c0f5410 2024-02-19T22:20:45.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:45.586+0000 7f4a74183700 1 -- 172.21.15.88:0/3550597007 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4a6c0fc690 con 0x7f4a6c0f5410 2024-02-19T22:20:45.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:45.586+0000 7f4a627fc700 1 -- 172.21.15.88:0/3550597007 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4a5c02b020 con 0x7f4a6c0f5410 2024-02-19T22:20:45.589 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:20:45.589 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:45.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:45.588+0000 7f4a74183700 1 -- 172.21.15.88:0/3550597007 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f4a5802ff10 msgr2=0x7f4a580323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:45.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:45.588+0000 7f4a74183700 1 --2- 172.21.15.88:0/3550597007 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f4a5802ff10 0x7f4a580323d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f4a680096a0 tx=0x7f4a68006b40).stop 2024-02-19T22:20:45.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:45.588+0000 7f4a74183700 1 -- 172.21.15.88:0/3550597007 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4a6c0f5410 msgr2=0x7f4a6c0fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:45.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:45.588+0000 7f4a74183700 1 --2- 172.21.15.88:0/3550597007 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4a6c0f5410 0x7f4a6c0fd4e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f4a5c005010 tx=0x7f4a5c00b040).stop 2024-02-19T22:20:45.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:45.588+0000 7f4a74183700 1 -- 172.21.15.88:0/3550597007 shutdown_connections 2024-02-19T22:20:45.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:45.588+0000 7f4a74183700 1 --2- 172.21.15.88:0/3550597007 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f4a5802ff10 0x7f4a580323d0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:45.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:45.588+0000 7f4a74183700 1 --2- 172.21.15.88:0/3550597007 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4a6c0f5410 0x7f4a6c0fd4e0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:45.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:45.588+0000 7f4a74183700 1 -- 172.21.15.88:0/3550597007 >> 172.21.15.88:0/3550597007 conn(0x7f4a6c0f0b70 msgr2=0x7f4a6c05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:45.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:45.588+0000 7f4a74183700 1 -- 172.21.15.88:0/3550597007 shutdown_connections 2024-02-19T22:20:45.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:45.588+0000 7f4a74183700 1 -- 172.21.15.88:0/3550597007 wait complete. 2024-02-19T22:20:45.591 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:20:46.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:20:44.599035+0000 mgr.smithi052.ghqrtk (mgr.14162) 341 : cluster [DBG] pgmap v325: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:46.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:20:45.587825+0000 mon.smithi052 (mon.0) 324 : audit [DBG] from='client.? 172.21.15.88:0/3550597007' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:47.230 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:47.230 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:48.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:48 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:20:46.599367+0000 mgr.smithi052.ghqrtk (mgr.14162) 342 : cluster [DBG] pgmap v326: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:49.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:49.092+0000 7f42e474f700 1 -- 172.21.15.88:0/1970160167 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f42dc0f6510 msgr2=0x7f42dc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:49.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:49.092+0000 7f42e474f700 1 --2- 172.21.15.88:0/1970160167 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f42dc0f6510 0x7f42dc0f6930 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f42cc009fb0 tx=0x7f42cc009b30).stop 2024-02-19T22:20:49.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:49.092+0000 7f42e474f700 1 -- 172.21.15.88:0/1970160167 shutdown_connections 2024-02-19T22:20:49.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:49.092+0000 7f42e474f700 1 --2- 172.21.15.88:0/1970160167 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f42dc0f6510 0x7f42dc0f6930 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:49.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:49.092+0000 7f42e474f700 1 -- 172.21.15.88:0/1970160167 >> 172.21.15.88:0/1970160167 conn(0x7f42dc0f1c70 msgr2=0x7f42dc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:49.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:49.092+0000 7f42e474f700 1 -- 172.21.15.88:0/1970160167 shutdown_connections 2024-02-19T22:20:49.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:49.092+0000 7f42e474f700 1 -- 172.21.15.88:0/1970160167 wait complete. 2024-02-19T22:20:49.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:49.093+0000 7f42e474f700 1 Processor -- start 2024-02-19T22:20:49.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:49.093+0000 7f42e474f700 1 -- start start 2024-02-19T22:20:49.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:49.093+0000 7f42e474f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f42dc0f6510 0x7f42dc06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:49.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:49.093+0000 7f42e474f700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42dc069e20 con 0x7f42dc0f6510 2024-02-19T22:20:49.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:49.093+0000 7f42e24eb700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f42dc0f6510 0x7f42dc06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:49.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:49.093+0000 7f42e24eb700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f42dc0f6510 0x7f42dc06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:58700/0 (socket says 172.21.15.88:58700) 2024-02-19T22:20:49.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:49.093+0000 7f42e24eb700 1 -- 172.21.15.88:0/1014939899 learned_addr learned my addr 172.21.15.88:0/1014939899 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:20:49.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:49.093+0000 7f42e24eb700 1 -- 172.21.15.88:0/1014939899 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42cc005040 con 0x7f42dc0f6510 2024-02-19T22:20:49.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:49.093+0000 7f42e24eb700 1 --2- 172.21.15.88:0/1014939899 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f42dc0f6510 0x7f42dc06b9a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f42cc005010 tx=0x7f42cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:49.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:49.094+0000 7f42d2ffd700 1 -- 172.21.15.88:0/1014939899 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42cc009320 con 0x7f42dc0f6510 2024-02-19T22:20:49.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:49.094+0000 7f42e474f700 1 -- 172.21.15.88:0/1014939899 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42dc069fc0 con 0x7f42dc0f6510 2024-02-19T22:20:49.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:49.094+0000 7f42d2ffd700 1 -- 172.21.15.88:0/1014939899 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f42cc013380 con 0x7f42dc0f6510 2024-02-19T22:20:49.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:49.094+0000 7f42d2ffd700 1 -- 172.21.15.88:0/1014939899 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42cc014e50 con 0x7f42dc0f6510 2024-02-19T22:20:49.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:49.094+0000 7f42e474f700 1 -- 172.21.15.88:0/1014939899 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42dc06a450 con 0x7f42dc0f6510 2024-02-19T22:20:49.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:49.095+0000 7f42d2ffd700 1 -- 172.21.15.88:0/1014939899 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f42cc0095f0 con 0x7f42dc0f6510 2024-02-19T22:20:49.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:49.095+0000 7f42e474f700 1 -- 172.21.15.88:0/1014939899 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42dc059100 con 0x7f42dc0f6510 2024-02-19T22:20:49.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:49.095+0000 7f42d2ffd700 1 --2- 172.21.15.88:0/1014939899 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f42c802ff10 0x7f42c80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:49.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:49.095+0000 7f42d2ffd700 1 -- 172.21.15.88:0/1014939899 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f42cc040e90 con 0x7f42dc0f6510 2024-02-19T22:20:49.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:49.095+0000 7f42e1cea700 1 --2- 172.21.15.88:0/1014939899 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f42c802ff10 0x7f42c80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:49.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:49.095+0000 7f42e1cea700 1 --2- 172.21.15.88:0/1014939899 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f42c802ff10 0x7f42c80323d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f42d80096a0 tx=0x7f42d8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:49.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:49.099+0000 7f42d2ffd700 1 -- 172.21.15.88:0/1014939899 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f42cc014420 con 0x7f42dc0f6510 2024-02-19T22:20:49.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:49.308+0000 7f42e474f700 1 -- 172.21.15.88:0/1014939899 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f42dc0fc0a0 con 0x7f42dc0f6510 2024-02-19T22:20:49.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:49.308+0000 7f42d2ffd700 1 -- 172.21.15.88:0/1014939899 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f42cc010230 con 0x7f42dc0f6510 2024-02-19T22:20:49.311 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:20:49.311 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:49.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:49.310+0000 7f42e474f700 1 -- 172.21.15.88:0/1014939899 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f42c802ff10 msgr2=0x7f42c80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:49.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:49.310+0000 7f42e474f700 1 --2- 172.21.15.88:0/1014939899 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f42c802ff10 0x7f42c80323d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f42d80096a0 tx=0x7f42d8006b40).stop 2024-02-19T22:20:49.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:49.310+0000 7f42e474f700 1 -- 172.21.15.88:0/1014939899 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f42dc0f6510 msgr2=0x7f42dc06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:49.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:49.310+0000 7f42e474f700 1 --2- 172.21.15.88:0/1014939899 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f42dc0f6510 0x7f42dc06b9a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f42cc005010 tx=0x7f42cc00b040).stop 2024-02-19T22:20:49.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:49.310+0000 7f42e474f700 1 -- 172.21.15.88:0/1014939899 shutdown_connections 2024-02-19T22:20:49.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:49.310+0000 7f42e474f700 1 --2- 172.21.15.88:0/1014939899 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f42c802ff10 0x7f42c80323d0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:49.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:49.310+0000 7f42e474f700 1 --2- 172.21.15.88:0/1014939899 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f42dc0f6510 0x7f42dc06b9a0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:49.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:49.310+0000 7f42e474f700 1 -- 172.21.15.88:0/1014939899 >> 172.21.15.88:0/1014939899 conn(0x7f42dc0f1c70 msgr2=0x7f42dc0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:49.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:49.310+0000 7f42e474f700 1 -- 172.21.15.88:0/1014939899 shutdown_connections 2024-02-19T22:20:49.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:49.310+0000 7f42e474f700 1 -- 172.21.15.88:0/1014939899 wait complete. 2024-02-19T22:20:49.313 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:20:50.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:50 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:20:48.599677+0000 mgr.smithi052.ghqrtk (mgr.14162) 343 : cluster [DBG] pgmap v327: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:50.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:50 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:20:49.309645+0000 mon.smithi052 (mon.0) 325 : audit [DBG] from='client.? 172.21.15.88:0/1014939899' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:50.983 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:50.983 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:52.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:52 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:20:50.599975+0000 mgr.smithi052.ghqrtk (mgr.14162) 344 : cluster [DBG] pgmap v328: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:52.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:52.980+0000 7fbcddfaa700 1 -- 172.21.15.88:0/1238720647 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbcd80f51a0 msgr2=0x7fbcd80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:52.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:52.980+0000 7fbcddfaa700 1 --2- 172.21.15.88:0/1238720647 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbcd80f51a0 0x7fbcd80f55c0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7fbccc009fb0 tx=0x7fbccc009b30).stop 2024-02-19T22:20:52.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:52.981+0000 7fbcddfaa700 1 -- 172.21.15.88:0/1238720647 shutdown_connections 2024-02-19T22:20:52.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:52.981+0000 7fbcddfaa700 1 --2- 172.21.15.88:0/1238720647 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbcd80f51a0 0x7fbcd80f55c0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:52.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:52.981+0000 7fbcddfaa700 1 -- 172.21.15.88:0/1238720647 >> 172.21.15.88:0/1238720647 conn(0x7fbcd80f09a0 msgr2=0x7fbcd80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:52.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:52.981+0000 7fbcddfaa700 1 -- 172.21.15.88:0/1238720647 shutdown_connections 2024-02-19T22:20:52.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:52.981+0000 7fbcddfaa700 1 -- 172.21.15.88:0/1238720647 wait complete. 2024-02-19T22:20:52.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:52.981+0000 7fbcddfaa700 1 Processor -- start 2024-02-19T22:20:52.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:52.981+0000 7fbcddfaa700 1 -- start start 2024-02-19T22:20:52.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:52.981+0000 7fbcddfaa700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbcd80f51a0 0x7fbcd81065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:52.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:52.981+0000 7fbcddfaa700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd8106ac0 con 0x7fbcd80f51a0 2024-02-19T22:20:52.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:52.982+0000 7fbcdcfa8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbcd80f51a0 0x7fbcd81065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:52.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:52.982+0000 7fbcdcfa8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbcd80f51a0 0x7fbcd81065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:58722/0 (socket says 172.21.15.88:58722) 2024-02-19T22:20:52.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:52.982+0000 7fbcdcfa8700 1 -- 172.21.15.88:0/2833087248 learned_addr learned my addr 172.21.15.88:0/2833087248 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:20:52.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:52.982+0000 7fbcdcfa8700 1 -- 172.21.15.88:0/2833087248 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbccc005040 con 0x7fbcd80f51a0 2024-02-19T22:20:52.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:52.982+0000 7fbcdcfa8700 1 --2- 172.21.15.88:0/2833087248 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbcd80f51a0 0x7fbcd81065d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7fbccc0089f0 tx=0x7fbccc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:52.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:52.982+0000 7fbcd57fa700 1 -- 172.21.15.88:0/2833087248 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbccc009320 con 0x7fbcd80f51a0 2024-02-19T22:20:52.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:52.982+0000 7fbcddfaa700 1 -- 172.21.15.88:0/2833087248 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcd8106cc0 con 0x7fbcd80f51a0 2024-02-19T22:20:52.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:52.982+0000 7fbcd57fa700 1 -- 172.21.15.88:0/2833087248 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbccc013380 con 0x7fbcd80f51a0 2024-02-19T22:20:52.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:52.982+0000 7fbcd57fa700 1 -- 172.21.15.88:0/2833087248 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbccc014e20 con 0x7fbcd80f51a0 2024-02-19T22:20:52.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:52.982+0000 7fbcddfaa700 1 -- 172.21.15.88:0/2833087248 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcd8107150 con 0x7fbcd80f51a0 2024-02-19T22:20:52.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:52.983+0000 7fbcd57fa700 1 -- 172.21.15.88:0/2833087248 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbccc014420 con 0x7fbcd80f51a0 2024-02-19T22:20:52.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:52.983+0000 7fbcddfaa700 1 -- 172.21.15.88:0/2833087248 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcd8045ad0 con 0x7fbcd80f51a0 2024-02-19T22:20:52.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:52.983+0000 7fbcd57fa700 1 --2- 172.21.15.88:0/2833087248 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fbcc802ff10 0x7fbcc80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:52.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:52.983+0000 7fbcd57fa700 1 -- 172.21.15.88:0/2833087248 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbccc040ff0 con 0x7fbcd80f51a0 2024-02-19T22:20:52.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:52.983+0000 7fbcd7fff700 1 --2- 172.21.15.88:0/2833087248 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fbcc802ff10 0x7fbcc80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:52.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:52.984+0000 7fbcd7fff700 1 --2- 172.21.15.88:0/2833087248 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fbcc802ff10 0x7fbcc80323d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fbcc40096a0 tx=0x7fbcc4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:52.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:52.988+0000 7fbcd57fa700 1 -- 172.21.15.88:0/2833087248 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbccc0095f0 con 0x7fbcd80f51a0 2024-02-19T22:20:53.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:53.197+0000 7fbcddfaa700 1 -- 172.21.15.88:0/2833087248 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbcd80fad30 con 0x7fbcd80f51a0 2024-02-19T22:20:53.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:53.197+0000 7fbcd57fa700 1 -- 172.21.15.88:0/2833087248 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbccc01db70 con 0x7fbcd80f51a0 2024-02-19T22:20:53.200 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:20:53.200 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:53.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:53.199+0000 7fbcddfaa700 1 -- 172.21.15.88:0/2833087248 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fbcc802ff10 msgr2=0x7fbcc80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:53.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:53.199+0000 7fbcddfaa700 1 --2- 172.21.15.88:0/2833087248 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fbcc802ff10 0x7fbcc80323d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fbcc40096a0 tx=0x7fbcc4006b40).stop 2024-02-19T22:20:53.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:53.199+0000 7fbcddfaa700 1 -- 172.21.15.88:0/2833087248 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbcd80f51a0 msgr2=0x7fbcd81065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:53.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:53.199+0000 7fbcddfaa700 1 --2- 172.21.15.88:0/2833087248 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbcd80f51a0 0x7fbcd81065d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7fbccc0089f0 tx=0x7fbccc00b040).stop 2024-02-19T22:20:53.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:53.199+0000 7fbcddfaa700 1 -- 172.21.15.88:0/2833087248 shutdown_connections 2024-02-19T22:20:53.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:53.199+0000 7fbcddfaa700 1 --2- 172.21.15.88:0/2833087248 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fbcc802ff10 0x7fbcc80323d0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:53.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:53.199+0000 7fbcddfaa700 1 --2- 172.21.15.88:0/2833087248 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbcd80f51a0 0x7fbcd81065d0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:53.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:53.199+0000 7fbcddfaa700 1 -- 172.21.15.88:0/2833087248 >> 172.21.15.88:0/2833087248 conn(0x7fbcd80f09a0 msgr2=0x7fbcd805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:53.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:53.199+0000 7fbcddfaa700 1 -- 172.21.15.88:0/2833087248 shutdown_connections 2024-02-19T22:20:53.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:53.199+0000 7fbcddfaa700 1 -- 172.21.15.88:0/2833087248 wait complete. 2024-02-19T22:20:53.203 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:20:53.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:53 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:20:53.198759+0000 mon.smithi052 (mon.0) 326 : audit [DBG] from='client.? 172.21.15.88:0/2833087248' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:54.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:54 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:20:52.600361+0000 mgr.smithi052.ghqrtk (mgr.14162) 345 : cluster [DBG] pgmap v329: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:54.943 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:54.944 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:56.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:56 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:20:54.600661+0000 mgr.smithi052.ghqrtk (mgr.14162) 346 : cluster [DBG] pgmap v330: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:56.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:56.738+0000 7fe12a863700 1 -- 172.21.15.88:0/3316747128 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe1240f5410 msgr2=0x7fe1240f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:56.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:56.738+0000 7fe12a863700 1 --2- 172.21.15.88:0/3316747128 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe1240f5410 0x7fe1240f5830 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7fe114009fb0 tx=0x7fe114009b30).stop 2024-02-19T22:20:56.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:56.738+0000 7fe12a863700 1 -- 172.21.15.88:0/3316747128 shutdown_connections 2024-02-19T22:20:56.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:56.738+0000 7fe12a863700 1 --2- 172.21.15.88:0/3316747128 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe1240f5410 0x7fe1240f5830 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:56.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:56.738+0000 7fe12a863700 1 -- 172.21.15.88:0/3316747128 >> 172.21.15.88:0/3316747128 conn(0x7fe1240f0b70 msgr2=0x7fe1240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:56.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:56.738+0000 7fe12a863700 1 -- 172.21.15.88:0/3316747128 shutdown_connections 2024-02-19T22:20:56.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:56.738+0000 7fe12a863700 1 -- 172.21.15.88:0/3316747128 wait complete. 2024-02-19T22:20:56.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:56.738+0000 7fe12a863700 1 Processor -- start 2024-02-19T22:20:56.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:56.739+0000 7fe12a863700 1 -- start start 2024-02-19T22:20:56.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:56.739+0000 7fe12a863700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe1240f5410 0x7fe124109530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:56.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:56.739+0000 7fe12a863700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe124060220 con 0x7fe1240f5410 2024-02-19T22:20:56.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:56.739+0000 7fe123fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe1240f5410 0x7fe124109530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:56.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:56.739+0000 7fe123fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe1240f5410 0x7fe124109530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:48278/0 (socket says 172.21.15.88:48278) 2024-02-19T22:20:56.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:56.739+0000 7fe123fff700 1 -- 172.21.15.88:0/1532050499 learned_addr learned my addr 172.21.15.88:0/1532050499 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:20:56.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:56.739+0000 7fe123fff700 1 -- 172.21.15.88:0/1532050499 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe114005040 con 0x7fe1240f5410 2024-02-19T22:20:56.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:56.739+0000 7fe123fff700 1 --2- 172.21.15.88:0/1532050499 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe1240f5410 0x7fe124109530 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fe114005010 tx=0x7fe11400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:56.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:56.740+0000 7fe120ff9700 1 -- 172.21.15.88:0/1532050499 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe114009320 con 0x7fe1240f5410 2024-02-19T22:20:56.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:56.740+0000 7fe12a863700 1 -- 172.21.15.88:0/1532050499 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe124060420 con 0x7fe1240f5410 2024-02-19T22:20:56.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:56.740+0000 7fe120ff9700 1 -- 172.21.15.88:0/1532050499 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe114013380 con 0x7fe1240f5410 2024-02-19T22:20:56.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:56.740+0000 7fe120ff9700 1 -- 172.21.15.88:0/1532050499 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe114014e50 con 0x7fe1240f5410 2024-02-19T22:20:56.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:56.740+0000 7fe12a863700 1 -- 172.21.15.88:0/1532050499 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe124109be0 con 0x7fe1240f5410 2024-02-19T22:20:56.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:56.740+0000 7fe120ff9700 1 -- 172.21.15.88:0/1532050499 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe1140263a0 con 0x7fe1240f5410 2024-02-19T22:20:56.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:56.741+0000 7fe120ff9700 1 --2- 172.21.15.88:0/1532050499 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fe10c02ff10 0x7fe10c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:56.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:56.741+0000 7fe120ff9700 1 -- 172.21.15.88:0/1532050499 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe114040f90 con 0x7fe1240f5410 2024-02-19T22:20:56.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:56.741+0000 7fe12a863700 1 -- 172.21.15.88:0/1532050499 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe124059100 con 0x7fe1240f5410 2024-02-19T22:20:56.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:56.741+0000 7fe1237fe700 1 --2- 172.21.15.88:0/1532050499 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fe10c02ff10 0x7fe10c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:56.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:56.741+0000 7fe1237fe700 1 --2- 172.21.15.88:0/1532050499 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fe10c02ff10 0x7fe10c0323d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fe1180096a0 tx=0x7fe118006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:56.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:56.746+0000 7fe120ff9700 1 -- 172.21.15.88:0/1532050499 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe114012920 con 0x7fe1240f5410 2024-02-19T22:20:56.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:56.949+0000 7fe12a863700 1 -- 172.21.15.88:0/1532050499 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe12410a290 con 0x7fe1240f5410 2024-02-19T22:20:56.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:56.950+0000 7fe120ff9700 1 -- 172.21.15.88:0/1532050499 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe11402b020 con 0x7fe1240f5410 2024-02-19T22:20:56.951 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:20:56.952 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:56.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:56.951+0000 7fe12a863700 1 -- 172.21.15.88:0/1532050499 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fe10c02ff10 msgr2=0x7fe10c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:56.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:56.951+0000 7fe12a863700 1 --2- 172.21.15.88:0/1532050499 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fe10c02ff10 0x7fe10c0323d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fe1180096a0 tx=0x7fe118006b40).stop 2024-02-19T22:20:56.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:56.951+0000 7fe12a863700 1 -- 172.21.15.88:0/1532050499 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe1240f5410 msgr2=0x7fe124109530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:56.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:56.951+0000 7fe12a863700 1 --2- 172.21.15.88:0/1532050499 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe1240f5410 0x7fe124109530 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fe114005010 tx=0x7fe11400b040).stop 2024-02-19T22:20:56.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:56.952+0000 7fe12a863700 1 -- 172.21.15.88:0/1532050499 shutdown_connections 2024-02-19T22:20:56.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:56.952+0000 7fe12a863700 1 --2- 172.21.15.88:0/1532050499 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fe10c02ff10 0x7fe10c0323d0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:56.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:56.952+0000 7fe12a863700 1 --2- 172.21.15.88:0/1532050499 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe1240f5410 0x7fe124109530 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:56.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:56.952+0000 7fe12a863700 1 -- 172.21.15.88:0/1532050499 >> 172.21.15.88:0/1532050499 conn(0x7fe1240f0b70 msgr2=0x7fe1240f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:56.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:56.952+0000 7fe12a863700 1 -- 172.21.15.88:0/1532050499 shutdown_connections 2024-02-19T22:20:56.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:20:56.952+0000 7fe12a863700 1 -- 172.21.15.88:0/1532050499 wait complete. 2024-02-19T22:20:56.954 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:20:57.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:57 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:20:56.950271+0000 mon.smithi052 (mon.0) 327 : audit [DBG] from='client.? 172.21.15.88:0/1532050499' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:58.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:20:58 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:20:56.601040+0000 mgr.smithi052.ghqrtk (mgr.14162) 347 : cluster [DBG] pgmap v331: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:58.736 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:58.736 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:21:00.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:21:00 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:20:58.601410+0000 mgr.smithi052.ghqrtk (mgr.14162) 348 : cluster [DBG] pgmap v332: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:00.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:00.757+0000 7f0fb61e8700 1 -- 172.21.15.88:0/1210558413 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0fb00f62a0 msgr2=0x7f0fb00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:00.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:00.757+0000 7f0fb61e8700 1 --2- 172.21.15.88:0/1210558413 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0fb00f62a0 0x7f0fb00f66c0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f0fa0009fb0 tx=0x7f0fa0009b30).stop 2024-02-19T22:21:00.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:00.757+0000 7f0fb61e8700 1 -- 172.21.15.88:0/1210558413 shutdown_connections 2024-02-19T22:21:00.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:00.757+0000 7f0fb61e8700 1 --2- 172.21.15.88:0/1210558413 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0fb00f62a0 0x7f0fb00f66c0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:00.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:00.757+0000 7f0fb61e8700 1 -- 172.21.15.88:0/1210558413 >> 172.21.15.88:0/1210558413 conn(0x7f0fb00f1aa0 msgr2=0x7f0fb00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:00.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:00.757+0000 7f0fb61e8700 1 -- 172.21.15.88:0/1210558413 shutdown_connections 2024-02-19T22:21:00.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:00.757+0000 7f0fb61e8700 1 -- 172.21.15.88:0/1210558413 wait complete. 2024-02-19T22:21:00.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:00.757+0000 7f0fb61e8700 1 Processor -- start 2024-02-19T22:21:00.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:00.757+0000 7f0fb61e8700 1 -- start start 2024-02-19T22:21:00.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:00.757+0000 7f0fb61e8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0fb00f62a0 0x7f0fb01087a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:00.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:00.757+0000 7f0fb61e8700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb0108c90 con 0x7f0fb00f62a0 2024-02-19T22:21:00.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:00.758+0000 7f0fb51e6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0fb00f62a0 0x7f0fb01087a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:00.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:00.758+0000 7f0fb51e6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0fb00f62a0 0x7f0fb01087a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:48302/0 (socket says 172.21.15.88:48302) 2024-02-19T22:21:00.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:00.758+0000 7f0fb51e6700 1 -- 172.21.15.88:0/289308745 learned_addr learned my addr 172.21.15.88:0/289308745 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:21:00.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:00.758+0000 7f0fb51e6700 1 -- 172.21.15.88:0/289308745 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fa0005040 con 0x7f0fb00f62a0 2024-02-19T22:21:00.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:00.758+0000 7f0fb51e6700 1 --2- 172.21.15.88:0/289308745 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0fb00f62a0 0x7f0fb01087a0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f0fa00089f0 tx=0x7f0fa000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:00.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:00.758+0000 7f0fa5ffb700 1 -- 172.21.15.88:0/289308745 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fa0009320 con 0x7f0fb00f62a0 2024-02-19T22:21:00.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:00.758+0000 7f0fb61e8700 1 -- 172.21.15.88:0/289308745 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fb0108e90 con 0x7f0fb00f62a0 2024-02-19T22:21:00.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:00.758+0000 7f0fa5ffb700 1 -- 172.21.15.88:0/289308745 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0fa0013380 con 0x7f0fb00f62a0 2024-02-19T22:21:00.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:00.758+0000 7f0fa5ffb700 1 -- 172.21.15.88:0/289308745 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fa0014e20 con 0x7f0fb00f62a0 2024-02-19T22:21:00.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:00.759+0000 7f0fb61e8700 1 -- 172.21.15.88:0/289308745 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fb0109320 con 0x7f0fb00f62a0 2024-02-19T22:21:00.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:00.759+0000 7f0fa5ffb700 1 -- 172.21.15.88:0/289308745 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0fa00263a0 con 0x7f0fb00f62a0 2024-02-19T22:21:00.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:00.759+0000 7f0fb61e8700 1 -- 172.21.15.88:0/289308745 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fb0045ad0 con 0x7f0fb00f62a0 2024-02-19T22:21:00.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:00.759+0000 7f0fa5ffb700 1 --2- 172.21.15.88:0/289308745 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f0f9c02ff10 0x7f0f9c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:00.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:00.760+0000 7f0fa5ffb700 1 -- 172.21.15.88:0/289308745 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0fa0041280 con 0x7f0fb00f62a0 2024-02-19T22:21:00.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:00.760+0000 7f0fb49e5700 1 --2- 172.21.15.88:0/289308745 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f0f9c02ff10 0x7f0f9c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:00.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:00.760+0000 7f0fb49e5700 1 --2- 172.21.15.88:0/289308745 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f0f9c02ff10 0x7f0f9c0323d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f0fac0096a0 tx=0x7f0fac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:00.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:00.764+0000 7f0fa5ffb700 1 -- 172.21.15.88:0/289308745 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0fa00126f0 con 0x7f0fb00f62a0 2024-02-19T22:21:00.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:00.969+0000 7f0fb61e8700 1 -- 172.21.15.88:0/289308745 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0fb010a940 con 0x7f0fb00f62a0 2024-02-19T22:21:00.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:00.969+0000 7f0fa5ffb700 1 -- 172.21.15.88:0/289308745 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0fa00126f0 con 0x7f0fb00f62a0 2024-02-19T22:21:00.971 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:21:00.971 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:21:00.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:00.971+0000 7f0fb61e8700 1 -- 172.21.15.88:0/289308745 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f0f9c02ff10 msgr2=0x7f0f9c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:00.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:00.971+0000 7f0fb61e8700 1 --2- 172.21.15.88:0/289308745 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f0f9c02ff10 0x7f0f9c0323d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f0fac0096a0 tx=0x7f0fac006b40).stop 2024-02-19T22:21:00.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:00.971+0000 7f0fb61e8700 1 -- 172.21.15.88:0/289308745 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0fb00f62a0 msgr2=0x7f0fb01087a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:00.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:00.971+0000 7f0fb61e8700 1 --2- 172.21.15.88:0/289308745 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0fb00f62a0 0x7f0fb01087a0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f0fa00089f0 tx=0x7f0fa000b040).stop 2024-02-19T22:21:00.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:00.971+0000 7f0fb61e8700 1 -- 172.21.15.88:0/289308745 shutdown_connections 2024-02-19T22:21:00.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:00.971+0000 7f0fb61e8700 1 --2- 172.21.15.88:0/289308745 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f0f9c02ff10 0x7f0f9c0323d0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:00.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:00.971+0000 7f0fb61e8700 1 --2- 172.21.15.88:0/289308745 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0fb00f62a0 0x7f0fb01087a0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:00.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:00.971+0000 7f0fb61e8700 1 -- 172.21.15.88:0/289308745 >> 172.21.15.88:0/289308745 conn(0x7f0fb00f1aa0 msgr2=0x7f0fb00f3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:00.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:00.971+0000 7f0fb61e8700 1 -- 172.21.15.88:0/289308745 shutdown_connections 2024-02-19T22:21:00.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:00.971+0000 7f0fb61e8700 1 -- 172.21.15.88:0/289308745 wait complete. 2024-02-19T22:21:00.974 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:21:01.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:21:01 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:21:00.969905+0000 mon.smithi052 (mon.0) 328 : audit [DBG] from='client.? 172.21.15.88:0/289308745' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:21:02.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:21:02 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:21:00.601779+0000 mgr.smithi052.ghqrtk (mgr.14162) 349 : cluster [DBG] pgmap v333: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:02.735 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:21:02.736 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:21:04.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:21:04 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:21:02.602209+0000 mgr.smithi052.ghqrtk (mgr.14162) 350 : cluster [DBG] pgmap v334: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:04.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:04.736+0000 7f3ffb0fd700 1 -- 172.21.15.88:0/2845040111 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3ff40f6510 msgr2=0x7f3ff40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:04.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:04.736+0000 7f3ffb0fd700 1 --2- 172.21.15.88:0/2845040111 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3ff40f6510 0x7f3ff40f6930 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f3fe8009fb0 tx=0x7f3fe8009b30).stop 2024-02-19T22:21:04.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:04.736+0000 7f3ffb0fd700 1 -- 172.21.15.88:0/2845040111 shutdown_connections 2024-02-19T22:21:04.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:04.736+0000 7f3ffb0fd700 1 --2- 172.21.15.88:0/2845040111 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3ff40f6510 0x7f3ff40f6930 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:04.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:04.736+0000 7f3ffb0fd700 1 -- 172.21.15.88:0/2845040111 >> 172.21.15.88:0/2845040111 conn(0x7f3ff40f1c70 msgr2=0x7f3ff40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:04.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:04.736+0000 7f3ffb0fd700 1 -- 172.21.15.88:0/2845040111 shutdown_connections 2024-02-19T22:21:04.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:04.736+0000 7f3ffb0fd700 1 -- 172.21.15.88:0/2845040111 wait complete. 2024-02-19T22:21:04.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:04.737+0000 7f3ffb0fd700 1 Processor -- start 2024-02-19T22:21:04.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:04.737+0000 7f3ffb0fd700 1 -- start start 2024-02-19T22:21:04.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:04.737+0000 7f3ffb0fd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3ff40f6510 0x7f3ff406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:04.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:04.737+0000 7f3ffb0fd700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff4069e20 con 0x7f3ff40f6510 2024-02-19T22:21:04.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:04.737+0000 7f3ff8e99700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3ff40f6510 0x7f3ff406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:04.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:04.737+0000 7f3ff8e99700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3ff40f6510 0x7f3ff406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:58128/0 (socket says 172.21.15.88:58128) 2024-02-19T22:21:04.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:04.737+0000 7f3ff8e99700 1 -- 172.21.15.88:0/994568676 learned_addr learned my addr 172.21.15.88:0/994568676 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:21:04.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:04.738+0000 7f3ff8e99700 1 -- 172.21.15.88:0/994568676 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe8005040 con 0x7f3ff40f6510 2024-02-19T22:21:04.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:04.738+0000 7f3ff8e99700 1 --2- 172.21.15.88:0/994568676 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3ff40f6510 0x7f3ff406b9a0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f3fe8005010 tx=0x7f3fe800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:04.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:04.738+0000 7f3ff17fa700 1 -- 172.21.15.88:0/994568676 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fe8009320 con 0x7f3ff40f6510 2024-02-19T22:21:04.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:04.738+0000 7f3ffb0fd700 1 -- 172.21.15.88:0/994568676 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ff4069fc0 con 0x7f3ff40f6510 2024-02-19T22:21:04.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:04.738+0000 7f3ff17fa700 1 -- 172.21.15.88:0/994568676 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3fe8013380 con 0x7f3ff40f6510 2024-02-19T22:21:04.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:04.738+0000 7f3ff17fa700 1 -- 172.21.15.88:0/994568676 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fe8014e50 con 0x7f3ff40f6510 2024-02-19T22:21:04.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:04.738+0000 7f3ffb0fd700 1 -- 172.21.15.88:0/994568676 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ff406a450 con 0x7f3ff40f6510 2024-02-19T22:21:04.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:04.739+0000 7f3ff17fa700 1 -- 172.21.15.88:0/994568676 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3fe80095f0 con 0x7f3ff40f6510 2024-02-19T22:21:04.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:04.739+0000 7f3ffb0fd700 1 -- 172.21.15.88:0/994568676 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ff4059100 con 0x7f3ff40f6510 2024-02-19T22:21:04.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:04.739+0000 7f3ff17fa700 1 --2- 172.21.15.88:0/994568676 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f3fe402ff10 0x7f3fe40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:04.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:04.739+0000 7f3ff17fa700 1 -- 172.21.15.88:0/994568676 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3fe8040e90 con 0x7f3ff40f6510 2024-02-19T22:21:04.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:04.739+0000 7f3ff3fff700 1 --2- 172.21.15.88:0/994568676 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f3fe402ff10 0x7f3fe40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:04.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:04.739+0000 7f3ff3fff700 1 --2- 172.21.15.88:0/994568676 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f3fe402ff10 0x7f3fe40323d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f3fe00096a0 tx=0x7f3fe0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:04.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:04.744+0000 7f3ff17fa700 1 -- 172.21.15.88:0/994568676 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3fe8014420 con 0x7f3ff40f6510 2024-02-19T22:21:04.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:04.948+0000 7f3ffb0fd700 1 -- 172.21.15.88:0/994568676 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3ff40fc0a0 con 0x7f3ff40f6510 2024-02-19T22:21:04.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:04.949+0000 7f3ff17fa700 1 -- 172.21.15.88:0/994568676 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3fe8010230 con 0x7f3ff40f6510 2024-02-19T22:21:04.951 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:21:04.951 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:21:04.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:04.951+0000 7f3ffb0fd700 1 -- 172.21.15.88:0/994568676 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f3fe402ff10 msgr2=0x7f3fe40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:04.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:04.951+0000 7f3ffb0fd700 1 --2- 172.21.15.88:0/994568676 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f3fe402ff10 0x7f3fe40323d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f3fe00096a0 tx=0x7f3fe0006b40).stop 2024-02-19T22:21:04.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:04.951+0000 7f3ffb0fd700 1 -- 172.21.15.88:0/994568676 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3ff40f6510 msgr2=0x7f3ff406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:04.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:04.951+0000 7f3ffb0fd700 1 --2- 172.21.15.88:0/994568676 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3ff40f6510 0x7f3ff406b9a0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f3fe8005010 tx=0x7f3fe800b040).stop 2024-02-19T22:21:04.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:04.951+0000 7f3ffb0fd700 1 -- 172.21.15.88:0/994568676 shutdown_connections 2024-02-19T22:21:04.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:04.951+0000 7f3ffb0fd700 1 --2- 172.21.15.88:0/994568676 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f3fe402ff10 0x7f3fe40323d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:04.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:04.951+0000 7f3ffb0fd700 1 --2- 172.21.15.88:0/994568676 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3ff40f6510 0x7f3ff406b9a0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:04.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:04.951+0000 7f3ffb0fd700 1 -- 172.21.15.88:0/994568676 >> 172.21.15.88:0/994568676 conn(0x7f3ff40f1c70 msgr2=0x7f3ff40f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:04.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:04.951+0000 7f3ffb0fd700 1 -- 172.21.15.88:0/994568676 shutdown_connections 2024-02-19T22:21:04.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:04.951+0000 7f3ffb0fd700 1 -- 172.21.15.88:0/994568676 wait complete. 2024-02-19T22:21:04.953 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:21:05.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:21:05 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:21:04.949658+0000 mon.smithi052 (mon.0) 329 : audit [DBG] from='client.? 172.21.15.88:0/994568676' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:21:06.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:21:06 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:21:04.602628+0000 mgr.smithi052.ghqrtk (mgr.14162) 351 : cluster [DBG] pgmap v335: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:06.651 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:21:06.651 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:21:08.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:08.466+0000 7f42eeb1b700 1 -- 172.21.15.88:0/1583605529 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f42e80f6510 msgr2=0x7f42e80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:08.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:08.466+0000 7f42eeb1b700 1 --2- 172.21.15.88:0/1583605529 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f42e80f6510 0x7f42e80f6930 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f42dc009fb0 tx=0x7f42dc009b30).stop 2024-02-19T22:21:08.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:08.466+0000 7f42eeb1b700 1 -- 172.21.15.88:0/1583605529 shutdown_connections 2024-02-19T22:21:08.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:08.466+0000 7f42eeb1b700 1 --2- 172.21.15.88:0/1583605529 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f42e80f6510 0x7f42e80f6930 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:08.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:08.466+0000 7f42eeb1b700 1 -- 172.21.15.88:0/1583605529 >> 172.21.15.88:0/1583605529 conn(0x7f42e80f1c70 msgr2=0x7f42e80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:08.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:08.466+0000 7f42eeb1b700 1 -- 172.21.15.88:0/1583605529 shutdown_connections 2024-02-19T22:21:08.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:08.466+0000 7f42eeb1b700 1 -- 172.21.15.88:0/1583605529 wait complete. 2024-02-19T22:21:08.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:08.467+0000 7f42eeb1b700 1 Processor -- start 2024-02-19T22:21:08.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:08.467+0000 7f42eeb1b700 1 -- start start 2024-02-19T22:21:08.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:08.467+0000 7f42eeb1b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f42e80f6510 0x7f42e81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:08.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:08.467+0000 7f42eeb1b700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e8106cd0 con 0x7f42e80f6510 2024-02-19T22:21:08.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:08.467+0000 7f42ec8b7700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f42e80f6510 0x7f42e81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:08.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:08.467+0000 7f42ec8b7700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f42e80f6510 0x7f42e81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:58154/0 (socket says 172.21.15.88:58154) 2024-02-19T22:21:08.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:08.467+0000 7f42ec8b7700 1 -- 172.21.15.88:0/2511338544 learned_addr learned my addr 172.21.15.88:0/2511338544 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:21:08.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:08.467+0000 7f42ec8b7700 1 -- 172.21.15.88:0/2511338544 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42dc005040 con 0x7f42e80f6510 2024-02-19T22:21:08.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:08.468+0000 7f42ec8b7700 1 --2- 172.21.15.88:0/2511338544 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f42e80f6510 0x7f42e81067e0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f42dc005010 tx=0x7f42dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:08.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:08.468+0000 7f42e57fa700 1 -- 172.21.15.88:0/2511338544 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42dc009320 con 0x7f42e80f6510 2024-02-19T22:21:08.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:08.468+0000 7f42eeb1b700 1 -- 172.21.15.88:0/2511338544 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42e8106ed0 con 0x7f42e80f6510 2024-02-19T22:21:08.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:08.468+0000 7f42e57fa700 1 -- 172.21.15.88:0/2511338544 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f42dc013380 con 0x7f42e80f6510 2024-02-19T22:21:08.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:08.468+0000 7f42e57fa700 1 -- 172.21.15.88:0/2511338544 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42dc014e50 con 0x7f42e80f6510 2024-02-19T22:21:08.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:08.468+0000 7f42eeb1b700 1 -- 172.21.15.88:0/2511338544 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42e8107360 con 0x7f42e80f6510 2024-02-19T22:21:08.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:08.468+0000 7f42e57fa700 1 -- 172.21.15.88:0/2511338544 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f42dc0095f0 con 0x7f42e80f6510 2024-02-19T22:21:08.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:08.468+0000 7f42eeb1b700 1 -- 172.21.15.88:0/2511338544 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42e8045ad0 con 0x7f42e80f6510 2024-02-19T22:21:08.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:08.469+0000 7f42e57fa700 1 --2- 172.21.15.88:0/2511338544 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f42d802ff10 0x7f42d80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:08.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:08.469+0000 7f42e57fa700 1 -- 172.21.15.88:0/2511338544 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f42dc040e90 con 0x7f42e80f6510 2024-02-19T22:21:08.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:08.469+0000 7f42e7fff700 1 --2- 172.21.15.88:0/2511338544 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f42d802ff10 0x7f42d80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:08.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:08.469+0000 7f42e7fff700 1 --2- 172.21.15.88:0/2511338544 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f42d802ff10 0x7f42d80323d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f42d40096a0 tx=0x7f42d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:08.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:08.473+0000 7f42e57fa700 1 -- 172.21.15.88:0/2511338544 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f42dc014420 con 0x7f42e80f6510 2024-02-19T22:21:08.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:21:08 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:21:06.602976+0000 mgr.smithi052.ghqrtk (mgr.14162) 352 : cluster [DBG] pgmap v336: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:08.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:08.683+0000 7f42eeb1b700 1 -- 172.21.15.88:0/2511338544 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f42e8108210 con 0x7f42e80f6510 2024-02-19T22:21:08.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:08.684+0000 7f42e57fa700 1 -- 172.21.15.88:0/2511338544 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f42dc010230 con 0x7f42e80f6510 2024-02-19T22:21:08.686 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:21:08.686 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:21:08.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:08.685+0000 7f42eeb1b700 1 -- 172.21.15.88:0/2511338544 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f42d802ff10 msgr2=0x7f42d80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:08.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:08.685+0000 7f42eeb1b700 1 --2- 172.21.15.88:0/2511338544 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f42d802ff10 0x7f42d80323d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f42d40096a0 tx=0x7f42d4006b40).stop 2024-02-19T22:21:08.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:08.685+0000 7f42eeb1b700 1 -- 172.21.15.88:0/2511338544 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f42e80f6510 msgr2=0x7f42e81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:08.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:08.685+0000 7f42eeb1b700 1 --2- 172.21.15.88:0/2511338544 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f42e80f6510 0x7f42e81067e0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f42dc005010 tx=0x7f42dc00b040).stop 2024-02-19T22:21:08.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:08.686+0000 7f42eeb1b700 1 -- 172.21.15.88:0/2511338544 shutdown_connections 2024-02-19T22:21:08.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:08.686+0000 7f42eeb1b700 1 --2- 172.21.15.88:0/2511338544 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f42d802ff10 0x7f42d80323d0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:08.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:08.686+0000 7f42eeb1b700 1 --2- 172.21.15.88:0/2511338544 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f42e80f6510 0x7f42e81067e0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:08.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:08.686+0000 7f42eeb1b700 1 -- 172.21.15.88:0/2511338544 >> 172.21.15.88:0/2511338544 conn(0x7f42e80f1c70 msgr2=0x7f42e80f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:08.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:08.686+0000 7f42eeb1b700 1 -- 172.21.15.88:0/2511338544 shutdown_connections 2024-02-19T22:21:08.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:08.686+0000 7f42eeb1b700 1 -- 172.21.15.88:0/2511338544 wait complete. 2024-02-19T22:21:08.688 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:21:09.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:21:09 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:21:08.684296+0000 mon.smithi052 (mon.0) 330 : audit [DBG] from='client.? 172.21.15.88:0/2511338544' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:21:10.441 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:21:10.442 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:21:10.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:21:10 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:21:08.603313+0000 mgr.smithi052.ghqrtk (mgr.14162) 353 : cluster [DBG] pgmap v337: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:12.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:12.275+0000 7f825c481700 1 -- 172.21.15.88:0/726425744 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82540f5410 msgr2=0x7f82540f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:12.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:12.275+0000 7f825c481700 1 --2- 172.21.15.88:0/726425744 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82540f5410 0x7f82540f5830 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f8250009fb0 tx=0x7f8250009b30).stop 2024-02-19T22:21:12.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:12.275+0000 7f825c481700 1 -- 172.21.15.88:0/726425744 shutdown_connections 2024-02-19T22:21:12.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:12.275+0000 7f825c481700 1 --2- 172.21.15.88:0/726425744 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82540f5410 0x7f82540f5830 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:12.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:12.275+0000 7f825c481700 1 -- 172.21.15.88:0/726425744 >> 172.21.15.88:0/726425744 conn(0x7f82540f0b70 msgr2=0x7f82540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:12.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:12.275+0000 7f825c481700 1 -- 172.21.15.88:0/726425744 shutdown_connections 2024-02-19T22:21:12.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:12.275+0000 7f825c481700 1 -- 172.21.15.88:0/726425744 wait complete. 2024-02-19T22:21:12.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:12.276+0000 7f825c481700 1 Processor -- start 2024-02-19T22:21:12.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:12.276+0000 7f825c481700 1 -- start start 2024-02-19T22:21:12.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:12.276+0000 7f825c481700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82540f5410 0x7f82541067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:12.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:12.276+0000 7f825c481700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8254106cc0 con 0x7f82540f5410 2024-02-19T22:21:12.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:12.276+0000 7f825a21d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82540f5410 0x7f82541067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:12.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:12.276+0000 7f825a21d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82540f5410 0x7f82541067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:58166/0 (socket says 172.21.15.88:58166) 2024-02-19T22:21:12.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:12.276+0000 7f825a21d700 1 -- 172.21.15.88:0/3891787537 learned_addr learned my addr 172.21.15.88:0/3891787537 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:21:12.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:12.276+0000 7f825a21d700 1 -- 172.21.15.88:0/3891787537 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8250005040 con 0x7f82540f5410 2024-02-19T22:21:12.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:12.277+0000 7f825a21d700 1 --2- 172.21.15.88:0/3891787537 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82540f5410 0x7f82541067d0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f8250005010 tx=0x7f825000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:12.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:12.277+0000 7f824affd700 1 -- 172.21.15.88:0/3891787537 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8250009320 con 0x7f82540f5410 2024-02-19T22:21:12.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:12.277+0000 7f824affd700 1 -- 172.21.15.88:0/3891787537 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8250013380 con 0x7f82540f5410 2024-02-19T22:21:12.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:12.277+0000 7f825c481700 1 -- 172.21.15.88:0/3891787537 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8254106ec0 con 0x7f82540f5410 2024-02-19T22:21:12.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:12.277+0000 7f824affd700 1 -- 172.21.15.88:0/3891787537 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8250014e50 con 0x7f82540f5410 2024-02-19T22:21:12.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:12.277+0000 7f825c481700 1 -- 172.21.15.88:0/3891787537 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8254107350 con 0x7f82540f5410 2024-02-19T22:21:12.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:12.278+0000 7f824affd700 1 -- 172.21.15.88:0/3891787537 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f82500263a0 con 0x7f82540f5410 2024-02-19T22:21:12.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:12.278+0000 7f824affd700 1 --2- 172.21.15.88:0/3891787537 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f824002ff10 0x7f82400323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:12.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:12.278+0000 7f824affd700 1 -- 172.21.15.88:0/3891787537 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8250040ea0 con 0x7f82540f5410 2024-02-19T22:21:12.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:12.278+0000 7f825c481700 1 -- 172.21.15.88:0/3891787537 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8254045ad0 con 0x7f82540f5410 2024-02-19T22:21:12.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:12.278+0000 7f8259a1c700 1 --2- 172.21.15.88:0/3891787537 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f824002ff10 0x7f82400323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:12.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:12.278+0000 7f8259a1c700 1 --2- 172.21.15.88:0/3891787537 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f824002ff10 0x7f82400323d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f82440096a0 tx=0x7f8244006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:12.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:12.283+0000 7f824affd700 1 -- 172.21.15.88:0/3891787537 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8250012920 con 0x7f82540f5410 2024-02-19T22:21:12.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:12.488+0000 7f825c481700 1 -- 172.21.15.88:0/3891787537 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8254108200 con 0x7f82540f5410 2024-02-19T22:21:12.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:12.489+0000 7f824affd700 1 -- 172.21.15.88:0/3891787537 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f825002b020 con 0x7f82540f5410 2024-02-19T22:21:12.492 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:21:12.492 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:21:12.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:12.490+0000 7f825c481700 1 -- 172.21.15.88:0/3891787537 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f824002ff10 msgr2=0x7f82400323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:12.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:12.490+0000 7f825c481700 1 --2- 172.21.15.88:0/3891787537 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f824002ff10 0x7f82400323d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f82440096a0 tx=0x7f8244006b40).stop 2024-02-19T22:21:12.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:12.490+0000 7f825c481700 1 -- 172.21.15.88:0/3891787537 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82540f5410 msgr2=0x7f82541067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:12.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:12.490+0000 7f825c481700 1 --2- 172.21.15.88:0/3891787537 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82540f5410 0x7f82541067d0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f8250005010 tx=0x7f825000b040).stop 2024-02-19T22:21:12.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:12.490+0000 7f825c481700 1 -- 172.21.15.88:0/3891787537 shutdown_connections 2024-02-19T22:21:12.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:12.490+0000 7f825c481700 1 --2- 172.21.15.88:0/3891787537 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f824002ff10 0x7f82400323d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:12.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:12.490+0000 7f825c481700 1 --2- 172.21.15.88:0/3891787537 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82540f5410 0x7f82541067d0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:12.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:12.491+0000 7f825c481700 1 -- 172.21.15.88:0/3891787537 >> 172.21.15.88:0/3891787537 conn(0x7f82540f0b70 msgr2=0x7f825405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:12.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:12.491+0000 7f825c481700 1 -- 172.21.15.88:0/3891787537 shutdown_connections 2024-02-19T22:21:12.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:12.491+0000 7f825c481700 1 -- 172.21.15.88:0/3891787537 wait complete. 2024-02-19T22:21:12.494 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:21:12.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:21:12 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:21:10.603644+0000 mgr.smithi052.ghqrtk (mgr.14162) 354 : cluster [DBG] pgmap v338: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:13.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:21:13 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:21:12.490269+0000 mon.smithi052 (mon.0) 331 : audit [DBG] from='client.? 172.21.15.88:0/3891787537' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:21:14.139 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:21:14.139 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:21:14.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:21:14 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:21:12.603903+0000 mgr.smithi052.ghqrtk (mgr.14162) 355 : cluster [DBG] pgmap v339: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:16.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:16.091+0000 7f62ef8a3700 1 -- 172.21.15.88:0/290585860 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62e80f6510 msgr2=0x7f62e80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:16.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:16.091+0000 7f62ef8a3700 1 --2- 172.21.15.88:0/290585860 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62e80f6510 0x7f62e80f6930 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f62e4009fb0 tx=0x7f62e4009b30).stop 2024-02-19T22:21:16.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:16.091+0000 7f62ef8a3700 1 -- 172.21.15.88:0/290585860 shutdown_connections 2024-02-19T22:21:16.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:16.091+0000 7f62ef8a3700 1 --2- 172.21.15.88:0/290585860 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62e80f6510 0x7f62e80f6930 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:16.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:16.091+0000 7f62ef8a3700 1 -- 172.21.15.88:0/290585860 >> 172.21.15.88:0/290585860 conn(0x7f62e80f1c70 msgr2=0x7f62e80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:16.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:16.091+0000 7f62ef8a3700 1 -- 172.21.15.88:0/290585860 shutdown_connections 2024-02-19T22:21:16.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:16.091+0000 7f62ef8a3700 1 -- 172.21.15.88:0/290585860 wait complete. 2024-02-19T22:21:16.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:16.092+0000 7f62ef8a3700 1 Processor -- start 2024-02-19T22:21:16.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:16.092+0000 7f62ef8a3700 1 -- start start 2024-02-19T22:21:16.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:16.092+0000 7f62ef8a3700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62e80f6510 0x7f62e81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:16.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:16.092+0000 7f62ef8a3700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e8106cd0 con 0x7f62e80f6510 2024-02-19T22:21:16.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:16.092+0000 7f62ed63f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62e80f6510 0x7f62e81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:16.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:16.092+0000 7f62ed63f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62e80f6510 0x7f62e81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:40694/0 (socket says 172.21.15.88:40694) 2024-02-19T22:21:16.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:16.092+0000 7f62ed63f700 1 -- 172.21.15.88:0/3993052944 learned_addr learned my addr 172.21.15.88:0/3993052944 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:21:16.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:16.092+0000 7f62ed63f700 1 -- 172.21.15.88:0/3993052944 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62e4005040 con 0x7f62e80f6510 2024-02-19T22:21:16.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:16.092+0000 7f62ed63f700 1 --2- 172.21.15.88:0/3993052944 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62e80f6510 0x7f62e81067e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f62e4005010 tx=0x7f62e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:16.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:16.093+0000 7f62ddffb700 1 -- 172.21.15.88:0/3993052944 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62e4009320 con 0x7f62e80f6510 2024-02-19T22:21:16.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:16.093+0000 7f62ef8a3700 1 -- 172.21.15.88:0/3993052944 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62e8106ed0 con 0x7f62e80f6510 2024-02-19T22:21:16.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:16.093+0000 7f62ddffb700 1 -- 172.21.15.88:0/3993052944 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f62e4013380 con 0x7f62e80f6510 2024-02-19T22:21:16.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:16.093+0000 7f62ddffb700 1 -- 172.21.15.88:0/3993052944 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62e4014e50 con 0x7f62e80f6510 2024-02-19T22:21:16.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:16.093+0000 7f62ef8a3700 1 -- 172.21.15.88:0/3993052944 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62e8107360 con 0x7f62e80f6510 2024-02-19T22:21:16.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:16.094+0000 7f62ddffb700 1 -- 172.21.15.88:0/3993052944 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f62e40263a0 con 0x7f62e80f6510 2024-02-19T22:21:16.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:16.094+0000 7f62ddffb700 1 --2- 172.21.15.88:0/3993052944 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f62d402ff10 0x7f62d40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:16.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:16.094+0000 7f62ddffb700 1 -- 172.21.15.88:0/3993052944 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f62e4040f70 con 0x7f62e80f6510 2024-02-19T22:21:16.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:16.094+0000 7f62ef8a3700 1 -- 172.21.15.88:0/3993052944 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62e8045ad0 con 0x7f62e80f6510 2024-02-19T22:21:16.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:16.094+0000 7f62ece3e700 1 --2- 172.21.15.88:0/3993052944 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f62d402ff10 0x7f62d40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:16.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:16.095+0000 7f62ece3e700 1 --2- 172.21.15.88:0/3993052944 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f62d402ff10 0x7f62d40323d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f62d80096a0 tx=0x7f62d8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:16.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:16.099+0000 7f62ddffb700 1 -- 172.21.15.88:0/3993052944 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f62e4012920 con 0x7f62e80f6510 2024-02-19T22:21:16.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:16.302+0000 7f62ef8a3700 1 -- 172.21.15.88:0/3993052944 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f62e81080a0 con 0x7f62e80f6510 2024-02-19T22:21:16.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:16.303+0000 7f62ddffb700 1 -- 172.21.15.88:0/3993052944 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f62e402b020 con 0x7f62e80f6510 2024-02-19T22:21:16.306 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:21:16.306 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:21:16.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:16.304+0000 7f62ef8a3700 1 -- 172.21.15.88:0/3993052944 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f62d402ff10 msgr2=0x7f62d40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:16.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:16.304+0000 7f62ef8a3700 1 --2- 172.21.15.88:0/3993052944 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f62d402ff10 0x7f62d40323d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f62d80096a0 tx=0x7f62d8006b40).stop 2024-02-19T22:21:16.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:16.304+0000 7f62ef8a3700 1 -- 172.21.15.88:0/3993052944 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62e80f6510 msgr2=0x7f62e81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:16.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:16.304+0000 7f62ef8a3700 1 --2- 172.21.15.88:0/3993052944 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62e80f6510 0x7f62e81067e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f62e4005010 tx=0x7f62e400b040).stop 2024-02-19T22:21:16.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:16.304+0000 7f62ef8a3700 1 -- 172.21.15.88:0/3993052944 shutdown_connections 2024-02-19T22:21:16.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:16.304+0000 7f62ef8a3700 1 --2- 172.21.15.88:0/3993052944 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f62d402ff10 0x7f62d40323d0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:16.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:16.304+0000 7f62ef8a3700 1 --2- 172.21.15.88:0/3993052944 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62e80f6510 0x7f62e81067e0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:16.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:16.304+0000 7f62ef8a3700 1 -- 172.21.15.88:0/3993052944 >> 172.21.15.88:0/3993052944 conn(0x7f62e80f1c70 msgr2=0x7f62e80f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:16.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:16.304+0000 7f62ef8a3700 1 -- 172.21.15.88:0/3993052944 shutdown_connections 2024-02-19T22:21:16.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:16.304+0000 7f62ef8a3700 1 -- 172.21.15.88:0/3993052944 wait complete. 2024-02-19T22:21:16.308 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:21:16.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:21:16 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:21:14.604248+0000 mgr.smithi052.ghqrtk (mgr.14162) 356 : cluster [DBG] pgmap v340: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:17.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:21:17 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:21:16.304398+0000 mon.smithi052 (mon.0) 332 : audit [DBG] from='client.? 172.21.15.88:0/3993052944' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:21:17.949 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:21:17.950 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:21:18.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:21:18 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:21:16.604610+0000 mgr.smithi052.ghqrtk (mgr.14162) 357 : cluster [DBG] pgmap v341: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:20.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:20.004+0000 7fdc89330700 1 -- 172.21.15.88:0/3763045176 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdc840f51a0 msgr2=0x7fdc840f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:20.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:20.004+0000 7fdc89330700 1 --2- 172.21.15.88:0/3763045176 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdc840f51a0 0x7fdc840f55c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7fdc74009fb0 tx=0x7fdc74009b30).stop 2024-02-19T22:21:20.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:20.004+0000 7fdc89330700 1 -- 172.21.15.88:0/3763045176 shutdown_connections 2024-02-19T22:21:20.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:20.004+0000 7fdc89330700 1 --2- 172.21.15.88:0/3763045176 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdc840f51a0 0x7fdc840f55c0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:20.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:20.004+0000 7fdc89330700 1 -- 172.21.15.88:0/3763045176 >> 172.21.15.88:0/3763045176 conn(0x7fdc840f09a0 msgr2=0x7fdc840f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:20.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:20.004+0000 7fdc89330700 1 -- 172.21.15.88:0/3763045176 shutdown_connections 2024-02-19T22:21:20.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:20.004+0000 7fdc89330700 1 -- 172.21.15.88:0/3763045176 wait complete. 2024-02-19T22:21:20.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:20.004+0000 7fdc89330700 1 Processor -- start 2024-02-19T22:21:20.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:20.005+0000 7fdc89330700 1 -- start start 2024-02-19T22:21:20.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:20.005+0000 7fdc89330700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdc840f51a0 0x7fdc841065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:20.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:20.005+0000 7fdc89330700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc84106ac0 con 0x7fdc840f51a0 2024-02-19T22:21:20.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:20.005+0000 7fdc83fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdc840f51a0 0x7fdc841065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:20.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:20.005+0000 7fdc83fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdc840f51a0 0x7fdc841065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:40712/0 (socket says 172.21.15.88:40712) 2024-02-19T22:21:20.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:20.005+0000 7fdc83fff700 1 -- 172.21.15.88:0/2865359461 learned_addr learned my addr 172.21.15.88:0/2865359461 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:21:20.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:20.005+0000 7fdc83fff700 1 -- 172.21.15.88:0/2865359461 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc74005040 con 0x7fdc840f51a0 2024-02-19T22:21:20.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:20.005+0000 7fdc83fff700 1 --2- 172.21.15.88:0/2865359461 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdc840f51a0 0x7fdc841065d0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7fdc740089f0 tx=0x7fdc7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:20.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:20.006+0000 7fdc80ff9700 1 -- 172.21.15.88:0/2865359461 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc74009320 con 0x7fdc840f51a0 2024-02-19T22:21:20.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:20.006+0000 7fdc89330700 1 -- 172.21.15.88:0/2865359461 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc84106cc0 con 0x7fdc840f51a0 2024-02-19T22:21:20.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:20.006+0000 7fdc80ff9700 1 -- 172.21.15.88:0/2865359461 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdc74013380 con 0x7fdc840f51a0 2024-02-19T22:21:20.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:20.006+0000 7fdc80ff9700 1 -- 172.21.15.88:0/2865359461 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc74014e20 con 0x7fdc840f51a0 2024-02-19T22:21:20.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:20.006+0000 7fdc89330700 1 -- 172.21.15.88:0/2865359461 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc84107150 con 0x7fdc840f51a0 2024-02-19T22:21:20.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:20.007+0000 7fdc80ff9700 1 -- 172.21.15.88:0/2865359461 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdc74014420 con 0x7fdc840f51a0 2024-02-19T22:21:20.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:20.007+0000 7fdc89330700 1 -- 172.21.15.88:0/2865359461 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc84045ad0 con 0x7fdc840f51a0 2024-02-19T22:21:20.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:20.007+0000 7fdc80ff9700 1 --2- 172.21.15.88:0/2865359461 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdc6c02ff10 0x7fdc6c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:20.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:20.007+0000 7fdc80ff9700 1 -- 172.21.15.88:0/2865359461 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdc74040ff0 con 0x7fdc840f51a0 2024-02-19T22:21:20.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:20.007+0000 7fdc837fe700 1 --2- 172.21.15.88:0/2865359461 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdc6c02ff10 0x7fdc6c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:20.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:20.007+0000 7fdc837fe700 1 --2- 172.21.15.88:0/2865359461 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdc6c02ff10 0x7fdc6c0323d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fdc780096a0 tx=0x7fdc78006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:20.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:20.012+0000 7fdc80ff9700 1 -- 172.21.15.88:0/2865359461 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdc740095f0 con 0x7fdc840f51a0 2024-02-19T22:21:20.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:20.223+0000 7fdc89330700 1 -- 172.21.15.88:0/2865359461 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdc840fad30 con 0x7fdc840f51a0 2024-02-19T22:21:20.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:20.224+0000 7fdc80ff9700 1 -- 172.21.15.88:0/2865359461 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdc7401db70 con 0x7fdc840f51a0 2024-02-19T22:21:20.227 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:21:20.227 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:21:20.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:20.226+0000 7fdc89330700 1 -- 172.21.15.88:0/2865359461 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdc6c02ff10 msgr2=0x7fdc6c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:20.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:20.226+0000 7fdc89330700 1 --2- 172.21.15.88:0/2865359461 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdc6c02ff10 0x7fdc6c0323d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fdc780096a0 tx=0x7fdc78006b40).stop 2024-02-19T22:21:20.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:20.226+0000 7fdc89330700 1 -- 172.21.15.88:0/2865359461 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdc840f51a0 msgr2=0x7fdc841065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:20.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:20.226+0000 7fdc89330700 1 --2- 172.21.15.88:0/2865359461 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdc840f51a0 0x7fdc841065d0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7fdc740089f0 tx=0x7fdc7400b040).stop 2024-02-19T22:21:20.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:20.226+0000 7fdc89330700 1 -- 172.21.15.88:0/2865359461 shutdown_connections 2024-02-19T22:21:20.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:20.226+0000 7fdc89330700 1 --2- 172.21.15.88:0/2865359461 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fdc6c02ff10 0x7fdc6c0323d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:20.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:20.226+0000 7fdc89330700 1 --2- 172.21.15.88:0/2865359461 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdc840f51a0 0x7fdc841065d0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:20.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:20.226+0000 7fdc89330700 1 -- 172.21.15.88:0/2865359461 >> 172.21.15.88:0/2865359461 conn(0x7fdc840f09a0 msgr2=0x7fdc8405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:20.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:20.226+0000 7fdc89330700 1 -- 172.21.15.88:0/2865359461 shutdown_connections 2024-02-19T22:21:20.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:20.226+0000 7fdc89330700 1 -- 172.21.15.88:0/2865359461 wait complete. 2024-02-19T22:21:20.229 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:21:20.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:21:20 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:21:18.604958+0000 mgr.smithi052.ghqrtk (mgr.14162) 358 : cluster [DBG] pgmap v342: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:20.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:21:20 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:21:20.225369+0000 mon.smithi052 (mon.0) 333 : audit [DBG] from='client.? 172.21.15.88:0/2865359461' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:21:21.850 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:21:21.850 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:21:22.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:21:22 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:21:20.605265+0000 mgr.smithi052.ghqrtk (mgr.14162) 359 : cluster [DBG] pgmap v343: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:23.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:23.823+0000 7f6f97562700 1 -- 172.21.15.88:0/2639165196 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6f900f5410 msgr2=0x7f6f900f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:23.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:23.823+0000 7f6f97562700 1 --2- 172.21.15.88:0/2639165196 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6f900f5410 0x7f6f900f5830 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7f6f80009fb0 tx=0x7f6f80009b30).stop 2024-02-19T22:21:23.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:23.823+0000 7f6f97562700 1 -- 172.21.15.88:0/2639165196 shutdown_connections 2024-02-19T22:21:23.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:23.823+0000 7f6f97562700 1 --2- 172.21.15.88:0/2639165196 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6f900f5410 0x7f6f900f5830 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:23.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:23.823+0000 7f6f97562700 1 -- 172.21.15.88:0/2639165196 >> 172.21.15.88:0/2639165196 conn(0x7f6f900f0b70 msgr2=0x7f6f900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:23.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:23.823+0000 7f6f97562700 1 -- 172.21.15.88:0/2639165196 shutdown_connections 2024-02-19T22:21:23.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:23.823+0000 7f6f97562700 1 -- 172.21.15.88:0/2639165196 wait complete. 2024-02-19T22:21:23.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:23.824+0000 7f6f97562700 1 Processor -- start 2024-02-19T22:21:23.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:23.824+0000 7f6f97562700 1 -- start start 2024-02-19T22:21:23.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:23.824+0000 7f6f97562700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6f900f5410 0x7f6f90106800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:23.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:23.824+0000 7f6f97562700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f90106cf0 con 0x7f6f900f5410 2024-02-19T22:21:23.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:23.824+0000 7f6f952fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6f900f5410 0x7f6f90106800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:23.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:23.824+0000 7f6f952fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6f900f5410 0x7f6f90106800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:37196/0 (socket says 172.21.15.88:37196) 2024-02-19T22:21:23.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:23.824+0000 7f6f952fe700 1 -- 172.21.15.88:0/1441012018 learned_addr learned my addr 172.21.15.88:0/1441012018 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:21:23.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:23.824+0000 7f6f952fe700 1 -- 172.21.15.88:0/1441012018 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f80005040 con 0x7f6f900f5410 2024-02-19T22:21:23.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:23.824+0000 7f6f952fe700 1 --2- 172.21.15.88:0/1441012018 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6f900f5410 0x7f6f90106800 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f6f80005010 tx=0x7f6f8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:23.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:23.825+0000 7f6f85ffb700 1 -- 172.21.15.88:0/1441012018 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f80009320 con 0x7f6f900f5410 2024-02-19T22:21:23.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:23.825+0000 7f6f97562700 1 -- 172.21.15.88:0/1441012018 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f90106ef0 con 0x7f6f900f5410 2024-02-19T22:21:23.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:23.825+0000 7f6f85ffb700 1 -- 172.21.15.88:0/1441012018 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6f80013380 con 0x7f6f900f5410 2024-02-19T22:21:23.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:23.825+0000 7f6f85ffb700 1 -- 172.21.15.88:0/1441012018 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f80014e50 con 0x7f6f900f5410 2024-02-19T22:21:23.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:23.825+0000 7f6f97562700 1 -- 172.21.15.88:0/1441012018 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f90107380 con 0x7f6f900f5410 2024-02-19T22:21:23.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:23.826+0000 7f6f97562700 1 -- 172.21.15.88:0/1441012018 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f90045ad0 con 0x7f6f900f5410 2024-02-19T22:21:23.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:23.826+0000 7f6f85ffb700 1 -- 172.21.15.88:0/1441012018 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6f800095f0 con 0x7f6f900f5410 2024-02-19T22:21:23.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:23.826+0000 7f6f85ffb700 1 --2- 172.21.15.88:0/1441012018 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f6f7c02ff10 0x7f6f7c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:23.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:23.826+0000 7f6f85ffb700 1 -- 172.21.15.88:0/1441012018 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6f80040fa0 con 0x7f6f900f5410 2024-02-19T22:21:23.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:23.826+0000 7f6f94afd700 1 --2- 172.21.15.88:0/1441012018 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f6f7c02ff10 0x7f6f7c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:23.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:23.826+0000 7f6f94afd700 1 --2- 172.21.15.88:0/1441012018 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f6f7c02ff10 0x7f6f7c0323d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f6f8c0096a0 tx=0x7f6f8c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:23.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:23.830+0000 7f6f85ffb700 1 -- 172.21.15.88:0/1441012018 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6f80014420 con 0x7f6f900f5410 2024-02-19T22:21:24.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:24.037+0000 7f6f97562700 1 -- 172.21.15.88:0/1441012018 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6f90108230 con 0x7f6f900f5410 2024-02-19T22:21:24.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:24.037+0000 7f6f85ffb700 1 -- 172.21.15.88:0/1441012018 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6f80010230 con 0x7f6f900f5410 2024-02-19T22:21:24.040 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:21:24.040 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:21:24.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:24.039+0000 7f6f97562700 1 -- 172.21.15.88:0/1441012018 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f6f7c02ff10 msgr2=0x7f6f7c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:24.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:24.039+0000 7f6f97562700 1 --2- 172.21.15.88:0/1441012018 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f6f7c02ff10 0x7f6f7c0323d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f6f8c0096a0 tx=0x7f6f8c006b40).stop 2024-02-19T22:21:24.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:24.039+0000 7f6f97562700 1 -- 172.21.15.88:0/1441012018 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6f900f5410 msgr2=0x7f6f90106800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:24.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:24.039+0000 7f6f97562700 1 --2- 172.21.15.88:0/1441012018 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6f900f5410 0x7f6f90106800 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f6f80005010 tx=0x7f6f8000b040).stop 2024-02-19T22:21:24.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:24.039+0000 7f6f97562700 1 -- 172.21.15.88:0/1441012018 shutdown_connections 2024-02-19T22:21:24.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:24.039+0000 7f6f97562700 1 --2- 172.21.15.88:0/1441012018 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f6f7c02ff10 0x7f6f7c0323d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:24.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:24.039+0000 7f6f97562700 1 --2- 172.21.15.88:0/1441012018 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6f900f5410 0x7f6f90106800 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:24.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:24.039+0000 7f6f97562700 1 -- 172.21.15.88:0/1441012018 >> 172.21.15.88:0/1441012018 conn(0x7f6f900f0b70 msgr2=0x7f6f9005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:24.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:24.039+0000 7f6f97562700 1 -- 172.21.15.88:0/1441012018 shutdown_connections 2024-02-19T22:21:24.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:24.039+0000 7f6f97562700 1 -- 172.21.15.88:0/1441012018 wait complete. 2024-02-19T22:21:24.043 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:21:24.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:21:24 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:21:22.605589+0000 mgr.smithi052.ghqrtk (mgr.14162) 360 : cluster [DBG] pgmap v344: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:24.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:21:24 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:21:24.038792+0000 mon.smithi052 (mon.0) 334 : audit [DBG] from='client.? 172.21.15.88:0/1441012018' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:21:25.680 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:21:25.681 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:21:26.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:21:26 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:21:24.605951+0000 mgr.smithi052.ghqrtk (mgr.14162) 361 : cluster [DBG] pgmap v345: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:27.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:27.515+0000 7fe3f3c1a700 1 -- 172.21.15.88:0/3918420626 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3ec0f5410 msgr2=0x7fe3ec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:27.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:27.515+0000 7fe3f3c1a700 1 --2- 172.21.15.88:0/3918420626 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3ec0f5410 0x7fe3ec0f5830 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7fe3dc009fb0 tx=0x7fe3dc009b30).stop 2024-02-19T22:21:27.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:27.515+0000 7fe3f3c1a700 1 -- 172.21.15.88:0/3918420626 shutdown_connections 2024-02-19T22:21:27.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:27.515+0000 7fe3f3c1a700 1 --2- 172.21.15.88:0/3918420626 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3ec0f5410 0x7fe3ec0f5830 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:27.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:27.515+0000 7fe3f3c1a700 1 -- 172.21.15.88:0/3918420626 >> 172.21.15.88:0/3918420626 conn(0x7fe3ec0f0b70 msgr2=0x7fe3ec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:27.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:27.515+0000 7fe3f3c1a700 1 -- 172.21.15.88:0/3918420626 shutdown_connections 2024-02-19T22:21:27.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:27.515+0000 7fe3f3c1a700 1 -- 172.21.15.88:0/3918420626 wait complete. 2024-02-19T22:21:27.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:27.516+0000 7fe3f3c1a700 1 Processor -- start 2024-02-19T22:21:27.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:27.516+0000 7fe3f3c1a700 1 -- start start 2024-02-19T22:21:27.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:27.516+0000 7fe3f3c1a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3ec0f5410 0x7fe3ec109530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:27.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:27.516+0000 7fe3f3c1a700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ec060220 con 0x7fe3ec0f5410 2024-02-19T22:21:27.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:27.516+0000 7fe3f19b6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3ec0f5410 0x7fe3ec109530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:27.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:27.516+0000 7fe3f19b6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3ec0f5410 0x7fe3ec109530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:37208/0 (socket says 172.21.15.88:37208) 2024-02-19T22:21:27.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:27.516+0000 7fe3f19b6700 1 -- 172.21.15.88:0/151959949 learned_addr learned my addr 172.21.15.88:0/151959949 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:21:27.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:27.517+0000 7fe3f19b6700 1 -- 172.21.15.88:0/151959949 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3dc005040 con 0x7fe3ec0f5410 2024-02-19T22:21:27.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:27.517+0000 7fe3f19b6700 1 --2- 172.21.15.88:0/151959949 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3ec0f5410 0x7fe3ec109530 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7fe3dc005010 tx=0x7fe3dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:27.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:27.517+0000 7fe3e27fc700 1 -- 172.21.15.88:0/151959949 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3dc009320 con 0x7fe3ec0f5410 2024-02-19T22:21:27.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:27.517+0000 7fe3f3c1a700 1 -- 172.21.15.88:0/151959949 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3ec060420 con 0x7fe3ec0f5410 2024-02-19T22:21:27.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:27.517+0000 7fe3e27fc700 1 -- 172.21.15.88:0/151959949 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe3dc013380 con 0x7fe3ec0f5410 2024-02-19T22:21:27.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:27.517+0000 7fe3e27fc700 1 -- 172.21.15.88:0/151959949 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3dc014e50 con 0x7fe3ec0f5410 2024-02-19T22:21:27.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:27.517+0000 7fe3f3c1a700 1 -- 172.21.15.88:0/151959949 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3ec109be0 con 0x7fe3ec0f5410 2024-02-19T22:21:27.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:27.518+0000 7fe3f3c1a700 1 -- 172.21.15.88:0/151959949 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3ec059100 con 0x7fe3ec0f5410 2024-02-19T22:21:27.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:27.518+0000 7fe3e27fc700 1 -- 172.21.15.88:0/151959949 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe3dc0095f0 con 0x7fe3ec0f5410 2024-02-19T22:21:27.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:27.518+0000 7fe3e27fc700 1 --2- 172.21.15.88:0/151959949 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fe3d802ff10 0x7fe3d80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:27.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:27.518+0000 7fe3e27fc700 1 -- 172.21.15.88:0/151959949 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe3dc040e90 con 0x7fe3ec0f5410 2024-02-19T22:21:27.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:27.518+0000 7fe3f11b5700 1 --2- 172.21.15.88:0/151959949 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fe3d802ff10 0x7fe3d80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:27.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:27.519+0000 7fe3f11b5700 1 --2- 172.21.15.88:0/151959949 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fe3d802ff10 0x7fe3d80323d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fe3e80096a0 tx=0x7fe3e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:27.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:27.523+0000 7fe3e27fc700 1 -- 172.21.15.88:0/151959949 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe3dc014420 con 0x7fe3ec0f5410 2024-02-19T22:21:27.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:27.729+0000 7fe3f3c1a700 1 -- 172.21.15.88:0/151959949 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe3ec045ad0 con 0x7fe3ec0f5410 2024-02-19T22:21:27.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:27.729+0000 7fe3e27fc700 1 -- 172.21.15.88:0/151959949 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe3dc010230 con 0x7fe3ec0f5410 2024-02-19T22:21:27.732 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:21:27.732 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:21:27.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:27.731+0000 7fe3f3c1a700 1 -- 172.21.15.88:0/151959949 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fe3d802ff10 msgr2=0x7fe3d80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:27.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:27.731+0000 7fe3f3c1a700 1 --2- 172.21.15.88:0/151959949 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fe3d802ff10 0x7fe3d80323d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fe3e80096a0 tx=0x7fe3e8006b40).stop 2024-02-19T22:21:27.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:27.731+0000 7fe3f3c1a700 1 -- 172.21.15.88:0/151959949 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3ec0f5410 msgr2=0x7fe3ec109530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:27.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:27.731+0000 7fe3f3c1a700 1 --2- 172.21.15.88:0/151959949 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3ec0f5410 0x7fe3ec109530 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7fe3dc005010 tx=0x7fe3dc00b040).stop 2024-02-19T22:21:27.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:27.731+0000 7fe3f3c1a700 1 -- 172.21.15.88:0/151959949 shutdown_connections 2024-02-19T22:21:27.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:27.731+0000 7fe3f3c1a700 1 --2- 172.21.15.88:0/151959949 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fe3d802ff10 0x7fe3d80323d0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:27.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:27.731+0000 7fe3f3c1a700 1 --2- 172.21.15.88:0/151959949 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3ec0f5410 0x7fe3ec109530 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:27.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:27.731+0000 7fe3f3c1a700 1 -- 172.21.15.88:0/151959949 >> 172.21.15.88:0/151959949 conn(0x7fe3ec0f0b70 msgr2=0x7fe3ec0f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:27.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:27.731+0000 7fe3f3c1a700 1 -- 172.21.15.88:0/151959949 shutdown_connections 2024-02-19T22:21:27.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:27.731+0000 7fe3f3c1a700 1 -- 172.21.15.88:0/151959949 wait complete. 2024-02-19T22:21:27.734 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:21:28.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:21:28 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:21:26.606247+0000 mgr.smithi052.ghqrtk (mgr.14162) 362 : cluster [DBG] pgmap v346: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:28.594 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:21:28 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:21:27.730738+0000 mon.smithi052 (mon.0) 335 : audit [DBG] from='client.? 172.21.15.88:0/151959949' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:21:29.502 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:21:29.502 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:21:30.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:21:30 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:21:28.606617+0000 mgr.smithi052.ghqrtk (mgr.14162) 363 : cluster [DBG] pgmap v347: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:31.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:31.430+0000 7faf47f0f700 1 -- 172.21.15.88:0/2084280628 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf400f5410 msgr2=0x7faf400f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:31.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:31.430+0000 7faf47f0f700 1 --2- 172.21.15.88:0/2084280628 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf400f5410 0x7faf400f5830 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7faf30009fb0 tx=0x7faf30009b30).stop 2024-02-19T22:21:31.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:31.430+0000 7faf47f0f700 1 -- 172.21.15.88:0/2084280628 shutdown_connections 2024-02-19T22:21:31.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:31.430+0000 7faf47f0f700 1 --2- 172.21.15.88:0/2084280628 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf400f5410 0x7faf400f5830 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:31.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:31.430+0000 7faf47f0f700 1 -- 172.21.15.88:0/2084280628 >> 172.21.15.88:0/2084280628 conn(0x7faf400f0b70 msgr2=0x7faf400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:31.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:31.430+0000 7faf47f0f700 1 -- 172.21.15.88:0/2084280628 shutdown_connections 2024-02-19T22:21:31.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:31.430+0000 7faf47f0f700 1 -- 172.21.15.88:0/2084280628 wait complete. 2024-02-19T22:21:31.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:31.431+0000 7faf47f0f700 1 Processor -- start 2024-02-19T22:21:31.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:31.431+0000 7faf47f0f700 1 -- start start 2024-02-19T22:21:31.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:31.431+0000 7faf47f0f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf400f5410 0x7faf401094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:31.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:31.431+0000 7faf47f0f700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf40060240 con 0x7faf400f5410 2024-02-19T22:21:31.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:31.431+0000 7faf45cab700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf400f5410 0x7faf401094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:31.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:31.431+0000 7faf45cab700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf400f5410 0x7faf401094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:37226/0 (socket says 172.21.15.88:37226) 2024-02-19T22:21:31.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:31.431+0000 7faf45cab700 1 -- 172.21.15.88:0/2476365729 learned_addr learned my addr 172.21.15.88:0/2476365729 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:21:31.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:31.432+0000 7faf45cab700 1 -- 172.21.15.88:0/2476365729 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf30005040 con 0x7faf400f5410 2024-02-19T22:21:31.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:31.432+0000 7faf45cab700 1 --2- 172.21.15.88:0/2476365729 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf400f5410 0x7faf401094e0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7faf30005010 tx=0x7faf3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:31.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:31.432+0000 7faf367fc700 1 -- 172.21.15.88:0/2476365729 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf30009320 con 0x7faf400f5410 2024-02-19T22:21:31.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:31.432+0000 7faf47f0f700 1 -- 172.21.15.88:0/2476365729 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf40060440 con 0x7faf400f5410 2024-02-19T22:21:31.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:31.432+0000 7faf367fc700 1 -- 172.21.15.88:0/2476365729 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faf30013380 con 0x7faf400f5410 2024-02-19T22:21:31.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:31.432+0000 7faf367fc700 1 -- 172.21.15.88:0/2476365729 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf30014e50 con 0x7faf400f5410 2024-02-19T22:21:31.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:31.432+0000 7faf47f0f700 1 -- 172.21.15.88:0/2476365729 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf40109b90 con 0x7faf400f5410 2024-02-19T22:21:31.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:31.433+0000 7faf47f0f700 1 -- 172.21.15.88:0/2476365729 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf40045ad0 con 0x7faf400f5410 2024-02-19T22:21:31.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:31.433+0000 7faf367fc700 1 -- 172.21.15.88:0/2476365729 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faf300095f0 con 0x7faf400f5410 2024-02-19T22:21:31.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:31.433+0000 7faf367fc700 1 --2- 172.21.15.88:0/2476365729 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7faf2c02ff10 0x7faf2c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:31.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:31.433+0000 7faf367fc700 1 -- 172.21.15.88:0/2476365729 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faf30040f10 con 0x7faf400f5410 2024-02-19T22:21:31.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:31.433+0000 7faf454aa700 1 --2- 172.21.15.88:0/2476365729 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7faf2c02ff10 0x7faf2c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:31.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:31.434+0000 7faf454aa700 1 --2- 172.21.15.88:0/2476365729 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7faf2c02ff10 0x7faf2c0323d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7faf3c0096a0 tx=0x7faf3c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:31.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:31.438+0000 7faf367fc700 1 -- 172.21.15.88:0/2476365729 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faf30014420 con 0x7faf400f5410 2024-02-19T22:21:31.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:31.642+0000 7faf47f0f700 1 -- 172.21.15.88:0/2476365729 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faf4010a3b0 con 0x7faf400f5410 2024-02-19T22:21:31.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:31.643+0000 7faf367fc700 1 -- 172.21.15.88:0/2476365729 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7faf30010230 con 0x7faf400f5410 2024-02-19T22:21:31.645 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:21:31.645 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:21:31.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:31.644+0000 7faf47f0f700 1 -- 172.21.15.88:0/2476365729 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7faf2c02ff10 msgr2=0x7faf2c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:31.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:31.644+0000 7faf47f0f700 1 --2- 172.21.15.88:0/2476365729 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7faf2c02ff10 0x7faf2c0323d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7faf3c0096a0 tx=0x7faf3c006b40).stop 2024-02-19T22:21:31.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:31.644+0000 7faf47f0f700 1 -- 172.21.15.88:0/2476365729 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf400f5410 msgr2=0x7faf401094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:31.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:31.644+0000 7faf47f0f700 1 --2- 172.21.15.88:0/2476365729 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf400f5410 0x7faf401094e0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7faf30005010 tx=0x7faf3000b040).stop 2024-02-19T22:21:31.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:31.644+0000 7faf47f0f700 1 -- 172.21.15.88:0/2476365729 shutdown_connections 2024-02-19T22:21:31.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:31.644+0000 7faf47f0f700 1 --2- 172.21.15.88:0/2476365729 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7faf2c02ff10 0x7faf2c0323d0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:31.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:31.644+0000 7faf47f0f700 1 --2- 172.21.15.88:0/2476365729 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf400f5410 0x7faf401094e0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:31.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:31.644+0000 7faf47f0f700 1 -- 172.21.15.88:0/2476365729 >> 172.21.15.88:0/2476365729 conn(0x7faf400f0b70 msgr2=0x7faf400f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:31.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:31.644+0000 7faf47f0f700 1 -- 172.21.15.88:0/2476365729 shutdown_connections 2024-02-19T22:21:31.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:31.644+0000 7faf47f0f700 1 -- 172.21.15.88:0/2476365729 wait complete. 2024-02-19T22:21:31.648 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:21:32.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:21:32 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:21:30.606857+0000 mgr.smithi052.ghqrtk (mgr.14162) 364 : cluster [DBG] pgmap v348: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:32.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:21:32 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:21:31.644105+0000 mon.smithi052 (mon.0) 336 : audit [DBG] from='client.? 172.21.15.88:0/2476365729' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:21:33.368 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:21:33.369 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:21:34.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:21:34 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:21:32.607218+0000 mgr.smithi052.ghqrtk (mgr.14162) 365 : cluster [DBG] pgmap v349: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:35.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:35.366+0000 7f23317db700 1 -- 172.21.15.88:0/1629118064 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f232c0f5410 msgr2=0x7f232c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:35.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:35.366+0000 7f23317db700 1 --2- 172.21.15.88:0/1629118064 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f232c0f5410 0x7f232c0f5830 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f2314009fb0 tx=0x7f2314009b30).stop 2024-02-19T22:21:35.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:35.366+0000 7f23317db700 1 -- 172.21.15.88:0/1629118064 shutdown_connections 2024-02-19T22:21:35.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:35.366+0000 7f23317db700 1 --2- 172.21.15.88:0/1629118064 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f232c0f5410 0x7f232c0f5830 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:35.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:35.366+0000 7f23317db700 1 -- 172.21.15.88:0/1629118064 >> 172.21.15.88:0/1629118064 conn(0x7f232c0f0b70 msgr2=0x7f232c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:35.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:35.366+0000 7f23317db700 1 -- 172.21.15.88:0/1629118064 shutdown_connections 2024-02-19T22:21:35.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:35.366+0000 7f23317db700 1 -- 172.21.15.88:0/1629118064 wait complete. 2024-02-19T22:21:35.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:35.367+0000 7f23317db700 1 Processor -- start 2024-02-19T22:21:35.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:35.367+0000 7f23317db700 1 -- start start 2024-02-19T22:21:35.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:35.367+0000 7f23317db700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f232c0f5410 0x7f232c105720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:35.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:35.367+0000 7f23317db700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f232c105c10 con 0x7f232c0f5410 2024-02-19T22:21:35.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:35.367+0000 7f232affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f232c0f5410 0x7f232c105720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:35.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:35.367+0000 7f232affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f232c0f5410 0x7f232c105720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:33304/0 (socket says 172.21.15.88:33304) 2024-02-19T22:21:35.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:35.367+0000 7f232affd700 1 -- 172.21.15.88:0/2813218646 learned_addr learned my addr 172.21.15.88:0/2813218646 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:21:35.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:35.368+0000 7f232affd700 1 -- 172.21.15.88:0/2813218646 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2314005040 con 0x7f232c0f5410 2024-02-19T22:21:35.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:35.368+0000 7f232affd700 1 --2- 172.21.15.88:0/2813218646 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f232c0f5410 0x7f232c105720 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f2314005010 tx=0x7f231400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:35.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:35.368+0000 7f23237fe700 1 -- 172.21.15.88:0/2813218646 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2314009320 con 0x7f232c0f5410 2024-02-19T22:21:35.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:35.368+0000 7f23317db700 1 -- 172.21.15.88:0/2813218646 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f232c105e10 con 0x7f232c0f5410 2024-02-19T22:21:35.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:35.368+0000 7f23237fe700 1 -- 172.21.15.88:0/2813218646 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2314013380 con 0x7f232c0f5410 2024-02-19T22:21:35.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:35.368+0000 7f23237fe700 1 -- 172.21.15.88:0/2813218646 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2314014e50 con 0x7f232c0f5410 2024-02-19T22:21:35.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:35.368+0000 7f23317db700 1 -- 172.21.15.88:0/2813218646 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f232c1062a0 con 0x7f232c0f5410 2024-02-19T22:21:35.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:35.369+0000 7f23237fe700 1 -- 172.21.15.88:0/2813218646 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f23140095f0 con 0x7f232c0f5410 2024-02-19T22:21:35.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:35.369+0000 7f23317db700 1 -- 172.21.15.88:0/2813218646 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f232c045ad0 con 0x7f232c0f5410 2024-02-19T22:21:35.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:35.369+0000 7f23237fe700 1 --2- 172.21.15.88:0/2813218646 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f231802ff10 0x7f23180323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:35.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:35.369+0000 7f23237fe700 1 -- 172.21.15.88:0/2813218646 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2314040e90 con 0x7f232c0f5410 2024-02-19T22:21:35.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:35.369+0000 7f232a7fc700 1 --2- 172.21.15.88:0/2813218646 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f231802ff10 0x7f23180323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:35.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:35.370+0000 7f232a7fc700 1 --2- 172.21.15.88:0/2813218646 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f231802ff10 0x7f23180323d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f231c0096a0 tx=0x7f231c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:35.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:35.374+0000 7f23237fe700 1 -- 172.21.15.88:0/2813218646 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2314014420 con 0x7f232c0f5410 2024-02-19T22:21:35.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:35.580+0000 7f23317db700 1 -- 172.21.15.88:0/2813218646 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f232c107150 con 0x7f232c0f5410 2024-02-19T22:21:35.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:35.581+0000 7f23237fe700 1 -- 172.21.15.88:0/2813218646 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2314010230 con 0x7f232c0f5410 2024-02-19T22:21:35.583 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:21:35.583 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:21:35.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:35.582+0000 7f23317db700 1 -- 172.21.15.88:0/2813218646 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f231802ff10 msgr2=0x7f23180323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:35.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:35.582+0000 7f23317db700 1 --2- 172.21.15.88:0/2813218646 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f231802ff10 0x7f23180323d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f231c0096a0 tx=0x7f231c006b40).stop 2024-02-19T22:21:35.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:35.582+0000 7f23317db700 1 -- 172.21.15.88:0/2813218646 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f232c0f5410 msgr2=0x7f232c105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:35.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:35.582+0000 7f23317db700 1 --2- 172.21.15.88:0/2813218646 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f232c0f5410 0x7f232c105720 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f2314005010 tx=0x7f231400b040).stop 2024-02-19T22:21:35.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:35.582+0000 7f23317db700 1 -- 172.21.15.88:0/2813218646 shutdown_connections 2024-02-19T22:21:35.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:35.582+0000 7f23317db700 1 --2- 172.21.15.88:0/2813218646 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f231802ff10 0x7f23180323d0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:35.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:35.582+0000 7f23317db700 1 --2- 172.21.15.88:0/2813218646 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f232c0f5410 0x7f232c105720 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:35.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:35.582+0000 7f23317db700 1 -- 172.21.15.88:0/2813218646 >> 172.21.15.88:0/2813218646 conn(0x7f232c0f0b70 msgr2=0x7f232c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:35.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:35.582+0000 7f23317db700 1 -- 172.21.15.88:0/2813218646 shutdown_connections 2024-02-19T22:21:35.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:35.582+0000 7f23317db700 1 -- 172.21.15.88:0/2813218646 wait complete. 2024-02-19T22:21:35.586 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:21:36.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:21:36 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:21:34.607646+0000 mgr.smithi052.ghqrtk (mgr.14162) 366 : cluster [DBG] pgmap v350: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:36.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:21:36 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:21:35.582161+0000 mon.smithi052 (mon.0) 337 : audit [DBG] from='client.? 172.21.15.88:0/2813218646' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:21:37.195 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:21:37.196 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:21:38.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:21:38 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:21:36.608058+0000 mgr.smithi052.ghqrtk (mgr.14162) 367 : cluster [DBG] pgmap v351: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:39.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:39.132+0000 7f180203d700 1 -- 172.21.15.88:0/3038587153 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f17fc0f6510 msgr2=0x7f17fc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:39.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:39.132+0000 7f180203d700 1 --2- 172.21.15.88:0/3038587153 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f17fc0f6510 0x7f17fc0f6930 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f17ec009fb0 tx=0x7f17ec009b30).stop 2024-02-19T22:21:39.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:39.132+0000 7f180203d700 1 -- 172.21.15.88:0/3038587153 shutdown_connections 2024-02-19T22:21:39.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:39.132+0000 7f180203d700 1 --2- 172.21.15.88:0/3038587153 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f17fc0f6510 0x7f17fc0f6930 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:39.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:39.132+0000 7f180203d700 1 -- 172.21.15.88:0/3038587153 >> 172.21.15.88:0/3038587153 conn(0x7f17fc0f1c70 msgr2=0x7f17fc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:39.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:39.132+0000 7f180203d700 1 -- 172.21.15.88:0/3038587153 shutdown_connections 2024-02-19T22:21:39.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:39.132+0000 7f180203d700 1 -- 172.21.15.88:0/3038587153 wait complete. 2024-02-19T22:21:39.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:39.132+0000 7f180203d700 1 Processor -- start 2024-02-19T22:21:39.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:39.133+0000 7f180203d700 1 -- start start 2024-02-19T22:21:39.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:39.133+0000 7f180203d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f17fc0f6510 0x7f17fc1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:39.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:39.133+0000 7f180203d700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17fc106cd0 con 0x7f17fc0f6510 2024-02-19T22:21:39.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:39.133+0000 7f17fb7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f17fc0f6510 0x7f17fc1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:39.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:39.133+0000 7f17fb7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f17fc0f6510 0x7f17fc1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:33326/0 (socket says 172.21.15.88:33326) 2024-02-19T22:21:39.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:39.133+0000 7f17fb7fe700 1 -- 172.21.15.88:0/1462816551 learned_addr learned my addr 172.21.15.88:0/1462816551 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:21:39.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:39.133+0000 7f17fb7fe700 1 -- 172.21.15.88:0/1462816551 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17ec005040 con 0x7f17fc0f6510 2024-02-19T22:21:39.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:39.134+0000 7f17fb7fe700 1 --2- 172.21.15.88:0/1462816551 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f17fc0f6510 0x7f17fc1067e0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f17ec005010 tx=0x7f17ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:39.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:39.134+0000 7f17e3fff700 1 -- 172.21.15.88:0/1462816551 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17ec009320 con 0x7f17fc0f6510 2024-02-19T22:21:39.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:39.134+0000 7f180203d700 1 -- 172.21.15.88:0/1462816551 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17fc106ed0 con 0x7f17fc0f6510 2024-02-19T22:21:39.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:39.134+0000 7f17e3fff700 1 -- 172.21.15.88:0/1462816551 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f17ec013380 con 0x7f17fc0f6510 2024-02-19T22:21:39.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:39.134+0000 7f17e3fff700 1 -- 172.21.15.88:0/1462816551 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17ec014e50 con 0x7f17fc0f6510 2024-02-19T22:21:39.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:39.134+0000 7f180203d700 1 -- 172.21.15.88:0/1462816551 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17fc107360 con 0x7f17fc0f6510 2024-02-19T22:21:39.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:39.135+0000 7f17e3fff700 1 -- 172.21.15.88:0/1462816551 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f17ec014420 con 0x7f17fc0f6510 2024-02-19T22:21:39.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:39.135+0000 7f180203d700 1 -- 172.21.15.88:0/1462816551 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17fc045ad0 con 0x7f17fc0f6510 2024-02-19T22:21:39.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:39.135+0000 7f17e3fff700 1 --2- 172.21.15.88:0/1462816551 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f17e402ff10 0x7f17e40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:39.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:39.135+0000 7f17e3fff700 1 -- 172.21.15.88:0/1462816551 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f17ec040e60 con 0x7f17fc0f6510 2024-02-19T22:21:39.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:39.135+0000 7f17faffd700 1 --2- 172.21.15.88:0/1462816551 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f17e402ff10 0x7f17e40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:39.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:39.135+0000 7f17faffd700 1 --2- 172.21.15.88:0/1462816551 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f17e402ff10 0x7f17e40323d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f17f00096a0 tx=0x7f17f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:39.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:39.139+0000 7f17e3fff700 1 -- 172.21.15.88:0/1462816551 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f17ec0095f0 con 0x7f17fc0f6510 2024-02-19T22:21:39.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:39.343+0000 7f180203d700 1 -- 172.21.15.88:0/1462816551 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f17fc1081b0 con 0x7f17fc0f6510 2024-02-19T22:21:39.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:39.344+0000 7f17e3fff700 1 -- 172.21.15.88:0/1462816551 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f17ec01db70 con 0x7f17fc0f6510 2024-02-19T22:21:39.348 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:21:39.348 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:21:39.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:39.346+0000 7f180203d700 1 -- 172.21.15.88:0/1462816551 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f17e402ff10 msgr2=0x7f17e40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:39.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:39.346+0000 7f180203d700 1 --2- 172.21.15.88:0/1462816551 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f17e402ff10 0x7f17e40323d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f17f00096a0 tx=0x7f17f0006b40).stop 2024-02-19T22:21:39.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:39.346+0000 7f180203d700 1 -- 172.21.15.88:0/1462816551 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f17fc0f6510 msgr2=0x7f17fc1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:39.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:39.346+0000 7f180203d700 1 --2- 172.21.15.88:0/1462816551 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f17fc0f6510 0x7f17fc1067e0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f17ec005010 tx=0x7f17ec00b040).stop 2024-02-19T22:21:39.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:39.346+0000 7f180203d700 1 -- 172.21.15.88:0/1462816551 shutdown_connections 2024-02-19T22:21:39.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:39.346+0000 7f180203d700 1 --2- 172.21.15.88:0/1462816551 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7f17e402ff10 0x7f17e40323d0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:39.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:39.346+0000 7f180203d700 1 --2- 172.21.15.88:0/1462816551 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f17fc0f6510 0x7f17fc1067e0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:39.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:39.346+0000 7f180203d700 1 -- 172.21.15.88:0/1462816551 >> 172.21.15.88:0/1462816551 conn(0x7f17fc0f1c70 msgr2=0x7f17fc0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:39.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:39.346+0000 7f180203d700 1 -- 172.21.15.88:0/1462816551 shutdown_connections 2024-02-19T22:21:39.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:39.346+0000 7f180203d700 1 -- 172.21.15.88:0/1462816551 wait complete. 2024-02-19T22:21:39.349 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:21:40.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:21:40 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:21:38.608491+0000 mgr.smithi052.ghqrtk (mgr.14162) 368 : cluster [DBG] pgmap v352: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:40.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:21:40 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:21:39.345609+0000 mon.smithi052 (mon.0) 338 : audit [DBG] from='client.? 172.21.15.88:0/1462816551' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:21:40.986 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:21:40.986 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:21:42.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:21:42 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:21:40.608831+0000 mgr.smithi052.ghqrtk (mgr.14162) 369 : cluster [DBG] pgmap v353: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:42.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:42.766+0000 7fa66759e700 1 -- 172.21.15.88:0/585831984 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa6680f78e0 msgr2=0x7fa6680f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:42.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:42.766+0000 7fa66759e700 1 --2- 172.21.15.88:0/585831984 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa6680f78e0 0x7fa6680f7d00 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7fa658009fb0 tx=0x7fa658009b30).stop 2024-02-19T22:21:42.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:42.766+0000 7fa66759e700 1 -- 172.21.15.88:0/585831984 shutdown_connections 2024-02-19T22:21:42.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:42.766+0000 7fa66759e700 1 --2- 172.21.15.88:0/585831984 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa6680f78e0 0x7fa6680f7d00 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:42.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:42.766+0000 7fa66759e700 1 -- 172.21.15.88:0/585831984 >> 172.21.15.88:0/585831984 conn(0x7fa6680f0120 msgr2=0x7fa6680f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:42.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:42.766+0000 7fa66759e700 1 -- 172.21.15.88:0/585831984 shutdown_connections 2024-02-19T22:21:42.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:42.766+0000 7fa66759e700 1 -- 172.21.15.88:0/585831984 wait complete. 2024-02-19T22:21:42.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:42.767+0000 7fa66759e700 1 Processor -- start 2024-02-19T22:21:42.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:42.767+0000 7fa66759e700 1 -- start start 2024-02-19T22:21:42.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:42.767+0000 7fa66759e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa6680f78e0 0x7fa6680f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:42.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:42.767+0000 7fa66759e700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6680f5630 con 0x7fa6680f78e0 2024-02-19T22:21:42.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:42.767+0000 7fa66659c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa6680f78e0 0x7fa6680f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:42.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:42.767+0000 7fa66659c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa6680f78e0 0x7fa6680f71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:33338/0 (socket says 172.21.15.88:33338) 2024-02-19T22:21:42.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:42.767+0000 7fa66659c700 1 -- 172.21.15.88:0/1478318551 learned_addr learned my addr 172.21.15.88:0/1478318551 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:21:42.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:42.768+0000 7fa66659c700 1 -- 172.21.15.88:0/1478318551 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa658005040 con 0x7fa6680f78e0 2024-02-19T22:21:42.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:42.768+0000 7fa66659c700 1 --2- 172.21.15.88:0/1478318551 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa6680f78e0 0x7fa6680f71b0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7fa6580089f0 tx=0x7fa65800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:42.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:42.768+0000 7fa65effd700 1 -- 172.21.15.88:0/1478318551 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa658009320 con 0x7fa6680f78e0 2024-02-19T22:21:42.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:42.768+0000 7fa66759e700 1 -- 172.21.15.88:0/1478318551 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6680f57d0 con 0x7fa6680f78e0 2024-02-19T22:21:42.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:42.768+0000 7fa65effd700 1 -- 172.21.15.88:0/1478318551 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa658013380 con 0x7fa6680f78e0 2024-02-19T22:21:42.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:42.768+0000 7fa65effd700 1 -- 172.21.15.88:0/1478318551 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa658014e20 con 0x7fa6680f78e0 2024-02-19T22:21:42.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:42.768+0000 7fa66759e700 1 -- 172.21.15.88:0/1478318551 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6680f5c60 con 0x7fa6680f78e0 2024-02-19T22:21:42.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:42.769+0000 7fa65effd700 1 -- 172.21.15.88:0/1478318551 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa658014420 con 0x7fa6680f78e0 2024-02-19T22:21:42.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:42.769+0000 7fa66759e700 1 -- 172.21.15.88:0/1478318551 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa668059100 con 0x7fa6680f78e0 2024-02-19T22:21:42.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:42.769+0000 7fa65effd700 1 --2- 172.21.15.88:0/1478318551 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa65402ff10 0x7fa6540323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:42.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:42.769+0000 7fa65effd700 1 -- 172.21.15.88:0/1478318551 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa658040f80 con 0x7fa6680f78e0 2024-02-19T22:21:42.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:42.769+0000 7fa665d9b700 1 --2- 172.21.15.88:0/1478318551 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa65402ff10 0x7fa6540323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:42.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:42.770+0000 7fa665d9b700 1 --2- 172.21.15.88:0/1478318551 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa65402ff10 0x7fa6540323d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fa6500096a0 tx=0x7fa650006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:42.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:42.775+0000 7fa65effd700 1 -- 172.21.15.88:0/1478318551 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa65806d2b0 con 0x7fa6680f78e0 2024-02-19T22:21:42.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:42.982+0000 7fa66759e700 1 -- 172.21.15.88:0/1478318551 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa6680fbe20 con 0x7fa6680f78e0 2024-02-19T22:21:42.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:42.982+0000 7fa65effd700 1 -- 172.21.15.88:0/1478318551 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa6580095f0 con 0x7fa6680f78e0 2024-02-19T22:21:42.985 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:21:42.985 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:21:42.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:42.984+0000 7fa66759e700 1 -- 172.21.15.88:0/1478318551 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa65402ff10 msgr2=0x7fa6540323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:42.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:42.984+0000 7fa66759e700 1 --2- 172.21.15.88:0/1478318551 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa65402ff10 0x7fa6540323d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fa6500096a0 tx=0x7fa650006b40).stop 2024-02-19T22:21:42.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:42.984+0000 7fa66759e700 1 -- 172.21.15.88:0/1478318551 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa6680f78e0 msgr2=0x7fa6680f71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:42.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:42.984+0000 7fa66759e700 1 --2- 172.21.15.88:0/1478318551 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa6680f78e0 0x7fa6680f71b0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7fa6580089f0 tx=0x7fa65800b040).stop 2024-02-19T22:21:42.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:42.984+0000 7fa66759e700 1 -- 172.21.15.88:0/1478318551 shutdown_connections 2024-02-19T22:21:42.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:42.984+0000 7fa66759e700 1 --2- 172.21.15.88:0/1478318551 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fa65402ff10 0x7fa6540323d0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:42.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:42.984+0000 7fa66759e700 1 --2- 172.21.15.88:0/1478318551 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa6680f78e0 0x7fa6680f71b0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:42.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:42.984+0000 7fa66759e700 1 -- 172.21.15.88:0/1478318551 >> 172.21.15.88:0/1478318551 conn(0x7fa6680f0120 msgr2=0x7fa6680f0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:42.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:42.984+0000 7fa66759e700 1 -- 172.21.15.88:0/1478318551 shutdown_connections 2024-02-19T22:21:42.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:42.984+0000 7fa66759e700 1 -- 172.21.15.88:0/1478318551 wait complete. 2024-02-19T22:21:42.987 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:21:43.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:21:43 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:21:42.983827+0000 mon.smithi052 (mon.0) 339 : audit [DBG] from='client.? 172.21.15.88:0/1478318551' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:21:44.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:21:44 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:21:42.609301+0000 mgr.smithi052.ghqrtk (mgr.14162) 370 : cluster [DBG] pgmap v354: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:44.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:21:44 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:21:43.757364+0000 mon.smithi052 (mon.0) 340 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi052.ghqrtk/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:21:44.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:21:44 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:21:43.759333+0000 mon.smithi052 (mon.0) 341 : audit [INF] from='mgr.14162 172.21.15.52:0/2682908552' entity='mgr.smithi052.ghqrtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi052.ghqrtk/trash_purge_schedule"}]: dispatch 2024-02-19T22:21:44.654 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:21:44.655 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 56739a54-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:21:46.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:21:46 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: cluster 2024-02-19T22:21:44.609622+0000 mgr.smithi052.ghqrtk (mgr.14162) 371 : cluster [DBG] pgmap v355: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:46.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:46.650+0000 7fd4d7915700 1 -- 172.21.15.88:0/2888527376 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd4d00f6510 msgr2=0x7fd4d00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:46.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:46.650+0000 7fd4d7915700 1 --2- 172.21.15.88:0/2888527376 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd4d00f6510 0x7fd4d00f6930 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7fd4cc009fb0 tx=0x7fd4cc009b30).stop 2024-02-19T22:21:46.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:46.650+0000 7fd4d7915700 1 -- 172.21.15.88:0/2888527376 shutdown_connections 2024-02-19T22:21:46.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:46.650+0000 7fd4d7915700 1 --2- 172.21.15.88:0/2888527376 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd4d00f6510 0x7fd4d00f6930 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:46.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:46.650+0000 7fd4d7915700 1 -- 172.21.15.88:0/2888527376 >> 172.21.15.88:0/2888527376 conn(0x7fd4d00f1c70 msgr2=0x7fd4d00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:46.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:46.650+0000 7fd4d7915700 1 -- 172.21.15.88:0/2888527376 shutdown_connections 2024-02-19T22:21:46.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:46.650+0000 7fd4d7915700 1 -- 172.21.15.88:0/2888527376 wait complete. 2024-02-19T22:21:46.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:46.651+0000 7fd4d7915700 1 Processor -- start 2024-02-19T22:21:46.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:46.651+0000 7fd4d7915700 1 -- start start 2024-02-19T22:21:46.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:46.651+0000 7fd4d7915700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd4d00f6510 0x7fd4d01067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:46.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:46.651+0000 7fd4d7915700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d0106cd0 con 0x7fd4d00f6510 2024-02-19T22:21:46.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:46.651+0000 7fd4d56b1700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd4d00f6510 0x7fd4d01067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:46.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:46.651+0000 7fd4d56b1700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd4d00f6510 0x7fd4d01067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.88:47624/0 (socket says 172.21.15.88:47624) 2024-02-19T22:21:46.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:46.651+0000 7fd4d56b1700 1 -- 172.21.15.88:0/644235108 learned_addr learned my addr 172.21.15.88:0/644235108 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-02-19T22:21:46.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:46.651+0000 7fd4d56b1700 1 -- 172.21.15.88:0/644235108 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4cc005040 con 0x7fd4d00f6510 2024-02-19T22:21:46.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:46.652+0000 7fd4d56b1700 1 --2- 172.21.15.88:0/644235108 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd4d00f6510 0x7fd4d01067e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7fd4cc005010 tx=0x7fd4cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:46.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:46.652+0000 7fd4c1ffb700 1 -- 172.21.15.88:0/644235108 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4cc009320 con 0x7fd4d00f6510 2024-02-19T22:21:46.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:46.652+0000 7fd4d7915700 1 -- 172.21.15.88:0/644235108 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4d0106ed0 con 0x7fd4d00f6510 2024-02-19T22:21:46.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:46.652+0000 7fd4c1ffb700 1 -- 172.21.15.88:0/644235108 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd4cc013380 con 0x7fd4d00f6510 2024-02-19T22:21:46.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:46.652+0000 7fd4c1ffb700 1 -- 172.21.15.88:0/644235108 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4cc014e50 con 0x7fd4d00f6510 2024-02-19T22:21:46.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:46.652+0000 7fd4d7915700 1 -- 172.21.15.88:0/644235108 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4d0107360 con 0x7fd4d00f6510 2024-02-19T22:21:46.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:46.653+0000 7fd4c1ffb700 1 -- 172.21.15.88:0/644235108 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd4cc0095f0 con 0x7fd4d00f6510 2024-02-19T22:21:46.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:46.653+0000 7fd4d7915700 1 -- 172.21.15.88:0/644235108 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4d0045ad0 con 0x7fd4d00f6510 2024-02-19T22:21:46.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:46.653+0000 7fd4c1ffb700 1 --2- 172.21.15.88:0/644235108 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd4bc02ff10 0x7fd4bc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:46.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:46.653+0000 7fd4c1ffb700 1 -- 172.21.15.88:0/644235108 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd4cc040e90 con 0x7fd4d00f6510 2024-02-19T22:21:46.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:46.653+0000 7fd4d4eb0700 1 --2- 172.21.15.88:0/644235108 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd4bc02ff10 0x7fd4bc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:46.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:46.654+0000 7fd4d4eb0700 1 --2- 172.21.15.88:0/644235108 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd4bc02ff10 0x7fd4bc0323d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fd4c40096a0 tx=0x7fd4c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:46.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:46.658+0000 7fd4c1ffb700 1 -- 172.21.15.88:0/644235108 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd4cc014420 con 0x7fd4d00f6510 2024-02-19T22:21:46.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:46.865+0000 7fd4d7915700 1 -- 172.21.15.88:0/644235108 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd4d01081b0 con 0x7fd4d00f6510 2024-02-19T22:21:46.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:46.865+0000 7fd4c1ffb700 1 -- 172.21.15.88:0/644235108 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd4cc010230 con 0x7fd4d00f6510 2024-02-19T22:21:46.867 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-02-19T22:21:46.867 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"56739a54-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:08:30.647471Z","created":"2024-02-19T22:08:30.647471Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi052","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:21:46.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:46.867+0000 7fd4d7915700 1 -- 172.21.15.88:0/644235108 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd4bc02ff10 msgr2=0x7fd4bc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:46.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:46.867+0000 7fd4d7915700 1 --2- 172.21.15.88:0/644235108 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd4bc02ff10 0x7fd4bc0323d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fd4c40096a0 tx=0x7fd4c4006b40).stop 2024-02-19T22:21:46.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:46.867+0000 7fd4d7915700 1 -- 172.21.15.88:0/644235108 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd4d00f6510 msgr2=0x7fd4d01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:46.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:46.867+0000 7fd4d7915700 1 --2- 172.21.15.88:0/644235108 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd4d00f6510 0x7fd4d01067e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7fd4cc005010 tx=0x7fd4cc00b040).stop 2024-02-19T22:21:46.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:46.867+0000 7fd4d7915700 1 -- 172.21.15.88:0/644235108 shutdown_connections 2024-02-19T22:21:46.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:46.867+0000 7fd4d7915700 1 --2- 172.21.15.88:0/644235108 >> [v2:172.21.15.52:6800/539827537,v1:172.21.15.52:6801/539827537] conn(0x7fd4bc02ff10 0x7fd4bc0323d0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:46.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:46.867+0000 7fd4d7915700 1 --2- 172.21.15.88:0/644235108 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd4d00f6510 0x7fd4d01067e0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:46.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:46.867+0000 7fd4d7915700 1 -- 172.21.15.88:0/644235108 >> 172.21.15.88:0/644235108 conn(0x7fd4d00f1c70 msgr2=0x7fd4d00f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:46.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:46.867+0000 7fd4d7915700 1 -- 172.21.15.88:0/644235108 shutdown_connections 2024-02-19T22:21:46.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-02-19T22:21:46.867+0000 7fd4d7915700 1 -- 172.21.15.88:0/644235108 wait complete. 2024-02-19T22:21:46.870 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-02-19T22:21:47.566 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds 2024-02-19T22:21:47.568 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-19T22:21:47.568 DEBUG:teuthology.orchestra.run.smithi052:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-19T22:21:47.593 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:21:47 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: audit 2024-02-19T22:21:46.865876+0000 mon.smithi052 (mon.0) 342 : audit [DBG] from='client.? 172.21.15.88:0/644235108' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:21:47.594 DEBUG:teuthology.orchestra.run.smithi088:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-19T22:21:47.612 INFO:tasks.cephadm:Stopping all daemons... 2024-02-19T22:21:47.613 INFO:tasks.cephadm.mon.smithi052:Stopping mon.smithi052... 2024-02-19T22:21:47.613 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ceph-56739a54-cf73-11ee-95bb-87774f69a715@mon.smithi052 2024-02-19T22:21:47.965 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:21:47 smithi052 systemd[1]: Stopping Ceph mon.smithi052 for 56739a54-cf73-11ee-95bb-87774f69a715... 2024-02-19T22:21:47.965 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:21:47 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:21:47.806+0000 7f69bf047700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi052 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-19T22:21:47.965 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:21:47 smithi052 ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052[38156]: debug 2024-02-19T22:21:47.806+0000 7f69bf047700 -1 mon.smithi052@0(leader) e1 *** Got Signal Terminated *** 2024-02-19T22:21:48.767 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 19 22:21:48 smithi052 podman[44217]: ceph-56739a54-cf73-11ee-95bb-87774f69a715-mon.smithi052 2024-02-19T22:21:48.910 DEBUG:teuthology.orchestra.run.smithi052:> sudo pkill -f 'journalctl -f -n 0 -u ceph-56739a54-cf73-11ee-95bb-87774f69a715@mon.smithi052.service' 2024-02-19T22:21:48.962 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-19T22:21:48.962 INFO:tasks.cephadm.mon.smithi052:Stopped mon.smithi052 2024-02-19T22:21:48.963 INFO:tasks.cephadm.mon.smithi088:Stopping mon.smithi088... 2024-02-19T22:21:48.963 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-56739a54-cf73-11ee-95bb-87774f69a715@mon.smithi088 2024-02-19T22:21:49.000 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to stop ceph-56739a54-cf73-11ee-95bb-87774f69a715@mon.smithi088.service: Unit ceph-56739a54-cf73-11ee-95bb-87774f69a715@mon.smithi088.service not loaded. 2024-02-19T22:21:49.001 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-02-19T22:21:49.002 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi088" Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 5: 'sudo systemctl stop ceph-56739a54-cf73-11ee-95bb-87774f69a715@mon.smithi088' 2024-02-19T22:21:49.003 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-19T22:21:49.004 DEBUG:teuthology.misc:Transferring archived files from smithi052:/var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566803/remote/ubuntu@smithi052.front.sepia.ceph.com/crash 2024-02-19T22:21:49.005 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar c -f - -C /var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/crash -- . 2024-02-19T22:21:49.072 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566803/remote/ubuntu@smithi088.front.sepia.ceph.com/crash 2024-02-19T22:21:49.073 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/crash -- . 2024-02-19T22:21:49.111 INFO:teuthology.orchestra.run.smithi088.stderr:tar: /var/lib/ceph/56739a54-cf73-11ee-95bb-87774f69a715/crash: Cannot open: No such file or directory 2024-02-19T22:21:49.111 INFO:teuthology.orchestra.run.smithi088.stderr:tar: Error is not recoverable: exiting now 2024-02-19T22:21:49.112 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-19T22:21:49.112 DEBUG:teuthology.orchestra.run.smithi052:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v 'Degraded data redundancy' | egrep -v '\(PG_' | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v 'deprecated feature inline_data' | egrep -v 'compat changed unexpectedly' | egrep -v 'overall HEALTH_' | egrep -v OSD_DOWN | egrep -v OSD_ | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-02-19T22:21:49.148 INFO:tasks.cephadm:Compressing logs... 2024-02-19T22:21:49.148 DEBUG:teuthology.orchestra.run.smithi052:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-19T22:21:49.190 DEBUG:teuthology.orchestra.run.smithi088:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-19T22:21:49.217 INFO:teuthology.orchestra.run.smithi052.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-19T22:21:49.217 INFO:teuthology.orchestra.run.smithi052.stderr:: No such file or directory 2024-02-19T22:21:49.217 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715/ceph-volume.log 2024-02-19T22:21:49.217 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715/ceph.audit.log 2024-02-19T22:21:49.218 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715/ceph.cephadm.log 2024-02-19T22:21:49.219 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715/ceph.log 2024-02-19T22:21:49.219 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715/ceph.audit.log: /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715/ceph-mgr.smithi052.ghqrtk.log 2024-02-19T22:21:49.219 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715/ceph.log: 90.2% -- replaced with /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715/ceph-volume.log.gz 2024-02-19T22:21:49.219 INFO:teuthology.orchestra.run.smithi052.stderr: 90.3% -- replaced with /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715/ceph.cephadm.log.gz 2024-02-19T22:21:49.219 INFO:teuthology.orchestra.run.smithi052.stderr: 88.8% -- replaced with /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715/ceph.audit.log.gz 2024-02-19T22:21:49.219 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715/ceph-mon.smithi052.log 2024-02-19T22:21:49.219 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715/ceph-mgr.smithi052.ghqrtk.log: 88.9% -- replaced with /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715/ceph.log.gz 2024-02-19T22:21:49.223 INFO:teuthology.orchestra.run.smithi088.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose : No such file or directory-- 2024-02-19T22:21:49.223 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/cephadm.log 2024-02-19T22:21:49.223 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715/ceph-volume.log 2024-02-19T22:21:49.224 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715/ceph-volume.log: 88.4% -- replaced with /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715/ceph-volume.log.gz 2024-02-19T22:21:49.225 INFO:teuthology.orchestra.run.smithi088.stderr: 95.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-19T22:21:49.226 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715/ceph-mon.smithi052.log: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-19T22:21:49.227 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-02-19T22:21:49.227 INFO:teuthology.orchestra.run.smithi088.stderr:real 0m0.016s 2024-02-19T22:21:49.227 INFO:teuthology.orchestra.run.smithi088.stderr:user 0m0.013s 2024-02-19T22:21:49.227 INFO:teuthology.orchestra.run.smithi088.stderr:sys 0m0.014s 2024-02-19T22:21:49.320 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715/ceph-mgr.smithi052.ghqrtk.log: file size changed while zipping 2024-02-19T22:21:49.320 INFO:teuthology.orchestra.run.smithi052.stderr: 92.7% -- replaced with /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715/ceph-mgr.smithi052.ghqrtk.log.gz 2024-02-19T22:21:49.397 INFO:teuthology.orchestra.run.smithi052.stderr: 90.6% -- replaced with /var/log/ceph/56739a54-cf73-11ee-95bb-87774f69a715/ceph-mon.smithi052.log.gz 2024-02-19T22:21:49.399 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-02-19T22:21:49.399 INFO:teuthology.orchestra.run.smithi052.stderr:real 0m0.193s 2024-02-19T22:21:49.399 INFO:teuthology.orchestra.run.smithi052.stderr:user 0m0.293s 2024-02-19T22:21:49.399 INFO:teuthology.orchestra.run.smithi052.stderr:sys 0m0.022s 2024-02-19T22:21:49.399 INFO:tasks.cephadm:Archiving logs... 2024-02-19T22:21:49.399 DEBUG:teuthology.misc:Transferring archived files from smithi052:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566803/remote/ubuntu@smithi052.front.sepia.ceph.com/log 2024-02-19T22:21:49.401 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-19T22:21:49.521 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566803/remote/ubuntu@smithi088.front.sepia.ceph.com/log 2024-02-19T22:21:49.522 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-19T22:21:49.556 INFO:tasks.cephadm:Removing cluster... 2024-02-19T22:21:49.557 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 56739a54-cf73-11ee-95bb-87774f69a715 --force 2024-02-19T22:21:53.124 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 56739a54-cf73-11ee-95bb-87774f69a715 --force 2024-02-19T22:24:50.272 INFO:tasks.cephadm:Removing cephadm ... 2024-02-19T22:24:50.272 DEBUG:teuthology.orchestra.run.smithi052:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-19T22:24:50.297 DEBUG:teuthology.orchestra.run.smithi088:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-19T22:24:50.315 INFO:tasks.cephadm:Teardown complete 2024-02-19T22:24:50.316 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 5: 'sudo systemctl stop ceph-56739a54-cf73-11ee-95bb-87774f69a715@mon.smithi088' 2024-02-19T22:24:50.555 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=15a517698ed54af988286b3da6d9eb03 Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 5: 'sudo systemctl stop ceph-56739a54-cf73-11ee-95bb-87774f69a715@mon.smithi088' 2024-02-19T22:24:50.560 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-19T22:24:50.570 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-19T22:24:50.579 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-19T22:24:50.579 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-19T22:24:50.603 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-19T22:24:50.699 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-19T22:24:50.708 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-19T22:24:50.708 DEBUG:teuthology.orchestra.run.smithi052:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-19T22:24:50.711 DEBUG:teuthology.orchestra.run.smithi088:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-19T22:24:50.728 INFO:teuthology.orchestra.run.smithi088.stderr:bash: ntpq: command not found 2024-02-19T22:24:50.730 INFO:teuthology.orchestra.run.smithi052.stderr:bash: ntpq: command not found 2024-02-19T22:24:50.732 INFO:teuthology.orchestra.run.smithi088.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-19T22:24:50.732 INFO:teuthology.orchestra.run.smithi088.stdout:=============================================================================== 2024-02-19T22:24:50.732 INFO:teuthology.orchestra.run.smithi088.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 26 +1827us[+1816us] +/- 103ms 2024-02-19T22:24:50.732 INFO:teuthology.orchestra.run.smithi088.stdout:^* hv02.front.sepia.ceph.com 3 7 377 26 -1611us[-1622us] +/- 56ms 2024-02-19T22:24:50.733 INFO:teuthology.orchestra.run.smithi088.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 28 +855us[ +844us] +/- 89ms 2024-02-19T22:24:50.733 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T22:24:50.735 INFO:teuthology.orchestra.run.smithi052.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-19T22:24:50.735 INFO:teuthology.orchestra.run.smithi052.stdout:=============================================================================== 2024-02-19T22:24:50.735 INFO:teuthology.orchestra.run.smithi052.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 28 +1867us[+1839us] +/- 103ms 2024-02-19T22:24:50.735 INFO:teuthology.orchestra.run.smithi052.stdout:^* hv02.front.sepia.ceph.com 3 6 377 26 -1590us[-1618us] +/- 56ms 2024-02-19T22:24:50.735 INFO:teuthology.orchestra.run.smithi052.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 25 +924us[ +924us] +/- 89ms 2024-02-19T22:24:50.736 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T22:24:50.737 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-19T22:24:50.746 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-19T22:24:50.747 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-19T22:24:50.826 DEBUG:teuthology.orchestra.run.smithi052:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-19T22:24:50.897 DEBUG:teuthology.orchestra.run.smithi088:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-19T22:24:50.960 DEBUG:teuthology.orchestra.run.smithi052:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-19T22:24:50.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T22:24:50.994 DEBUG:teuthology.orchestra.run.smithi088:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-19T22:24:51.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T22:24:51.023 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-19T22:24:51.032 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-19T22:24:51.068 INFO:teuthology.task.internal:Duration was 1764.624530 seconds 2024-02-19T22:24:51.069 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-19T22:24:51.121 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-19T22:24:51.122 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-19T22:24:51.127 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-19T22:24:51.173 INFO:teuthology.orchestra.run.smithi088.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-19T22:24:51.175 INFO:teuthology.orchestra.run.smithi052.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-19T22:24:51.357 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-19T22:24:51.358 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi052.front.sepia.ceph.com 2024-02-19T22:24:51.358 DEBUG:teuthology.orchestra.run.smithi052:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-19T22:24:51.384 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi088.front.sepia.ceph.com 2024-02-19T22:24:51.384 DEBUG:teuthology.orchestra.run.smithi088:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-19T22:24:51.406 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-19T22:24:51.406 DEBUG:teuthology.orchestra.run.smithi052:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-19T22:24:51.427 DEBUG:teuthology.orchestra.run.smithi088:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-19T22:24:51.496 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-19T22:24:51.497 DEBUG:teuthology.orchestra.run.smithi052:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-19T22:24:51.499 DEBUG:teuthology.orchestra.run.smithi088:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-19T22:24:51.589 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-19T22:24:51.599 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-19T22:24:51.599 DEBUG:teuthology.orchestra.run.smithi052:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-19T22:24:51.627 DEBUG:teuthology.orchestra.run.smithi088:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-19T22:24:51.653 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-19T22:24:51.663 DEBUG:teuthology.orchestra.run.smithi052:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-19T22:24:51.670 DEBUG:teuthology.orchestra.run.smithi088:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-19T22:24:51.696 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern = core 2024-02-19T22:24:51.720 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern = core 2024-02-19T22:24:51.744 DEBUG:teuthology.orchestra.run.smithi052:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-19T22:24:51.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T22:24:51.776 DEBUG:teuthology.orchestra.run.smithi088:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-19T22:24:51.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T22:24:51.801 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-19T22:24:51.811 INFO:teuthology.task.internal:Transferring archived files... 2024-02-19T22:24:51.812 DEBUG:teuthology.misc:Transferring archived files from smithi052:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566803/remote/smithi052 2024-02-19T22:24:51.813 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-19T22:24:51.857 DEBUG:teuthology.misc:Transferring archived files from smithi088:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566803/remote/smithi088 2024-02-19T22:24:51.858 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-19T22:24:51.898 INFO:teuthology.task.internal:Removing archive directory... 2024-02-19T22:24:51.898 DEBUG:teuthology.orchestra.run.smithi052:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-19T22:24:51.901 DEBUG:teuthology.orchestra.run.smithi088:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-19T22:24:51.948 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-19T22:24:51.959 INFO:teuthology.task.internal:Not uploading archives. 2024-02-19T22:24:51.960 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-19T22:24:52.023 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-19T22:24:52.023 DEBUG:teuthology.orchestra.run.smithi052:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-19T22:24:52.026 DEBUG:teuthology.orchestra.run.smithi088:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-19T22:24:52.041 INFO:teuthology.orchestra.run.smithi052.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 19 22:24 /home/ubuntu/cephtest 2024-02-19T22:24:52.043 INFO:teuthology.orchestra.run.smithi088.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 19 22:24 /home/ubuntu/cephtest 2024-02-19T22:24:52.044 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-19T22:24:52.054 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-19T22:24:52.126 INFO:teuthology.nuke:Checking targets against current locks 2024-02-19T22:24:52.161 DEBUG:teuthology.nuke:shortname: smithi052 2024-02-19T22:24:52.161 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-19T22:24:52.203 DEBUG:teuthology.nuke:shortname: smithi088 2024-02-19T22:24:52.203 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-19T22:24:52.216 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi052.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566803', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 21:46:04.763280', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK3gdrbO5novp6qfB1dH4FvuFEW8gqbmsgtUnuKo6FK5q5Z2YsmKPlBawQApgt+K1xXvoIYa9OXRPIL808vgEs4='} 2024-02-19T22:24:52.249 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi088.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566803', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 21:46:04.764852', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJsfHsnFkPbhk/aA3BZaiEoJv0bulu+9B1Y4Z9juUllAkaC2GW4G+K0qS9Zxw44kth7QvmHbxwlmvRgZ3K2bAes='} 2024-02-19T22:24:52.309 INFO:teuthology.orchestra.console.smithi052:Power off 2024-02-19T22:24:52.309 DEBUG:teuthology.orchestra.console.smithi052:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-19T22:24:52.348 INFO:teuthology.orchestra.console.smithi088:Power off 2024-02-19T22:24:52.349 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-19T22:24:52.363 DEBUG:teuthology.orchestra.console.smithi052:power off output: Chassis Power Control: Down/Off 2024-02-19T22:24:52.363 DEBUG:teuthology.orchestra.console.smithi052:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T22:24:52.396 DEBUG:teuthology.orchestra.console.smithi088:power off output: Chassis Power Control: Down/Off 2024-02-19T22:24:52.396 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T22:24:52.416 DEBUG:teuthology.orchestra.console.smithi052:check power output: Chassis Power is on 2024-02-19T22:24:52.444 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is on 2024-02-19T22:24:56.417 DEBUG:teuthology.orchestra.console.smithi052:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T22:24:56.445 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T22:24:56.529 DEBUG:teuthology.orchestra.console.smithi052:check power output: Chassis Power is on 2024-02-19T22:24:56.557 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is on 2024-02-19T22:25:00.534 DEBUG:teuthology.orchestra.console.smithi052:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T22:25:00.558 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T22:25:00.644 DEBUG:teuthology.orchestra.console.smithi052:check power output: Chassis Power is off 2024-02-19T22:25:00.668 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is off 2024-02-19T22:25:00.745 INFO:teuthology.orchestra.console.smithi052:Power off completed 2024-02-19T22:25:00.769 INFO:teuthology.orchestra.console.smithi088:Power off completed 2024-02-19T22:25:00.874 INFO:teuthology.run:Summary data: description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 1764.624529838562 failure_reason: 'Command failed on smithi088 with status 5: ''sudo systemctl stop ceph-56739a54-cf73-11ee-95bb-87774f69a715@mon.smithi088''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=15a517698ed54af988286b3da6d9eb03 status: fail success: false 2024-02-19T22:25:00.875 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-19T22:25:01.041 INFO:teuthology.run:FAIL