2024-01-21T19:44:51.425 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-21T19:44:51.425 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} --name yuriw-2024-01-21_16:37:18-fs-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2024-01-21_16:37:18-fs-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7525069 --verbose --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2024-01-21_16:37:18-fs-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7525069/orig.config.yaml 2024-01-21T19:44:51.473 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-21T19:44:51.572 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-01-21_16:37:18-fs-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7525069 branch: wip-yuri4-testing-2024-01-18-1257-pacific description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: null first_in_suite: false job_id: '7525069' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: yuriw-2024-01-21_16:37:18-fs-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri4-testing-2024-01-18-1257-pacific ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri4-testing-2024-01-18-1257-pacific sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 9457 sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec sleep_before_teardown: 0 subset: 111/12000 suite: fs suite_branch: wip-yuri4-testing-2024-01-18-1257-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec targets: smithi053.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKpOsOjovXAabvq3jBF/C3fX8dBGkOjGUi3rKmXmWHvlzOiZgPGWaB05cvinr0wkjK+mL2LXbbaLTbta922SPeA= smithi146.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD3QXMV9nD31+bSTFbZE+8Kvev917q1qJtUgAKnX7s/UtLVRVFfrd0QefK8rG/D6DvyvClaX7OrgeJnt77fF0UQ= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - pexec: clients: - sudo modprobe -r ceph - sudo modprobe ceph disable_send_metrics=on - install: exclude_packages: - ceph-volume tag: v16.2.4 - print: '**** done install task...' - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data false - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: cd45576300487d997e5a85abed65500b9f5d143b timestamp: 2024-01-21_16:37:18 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-01-21T19:44:51.572 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa; will attempt to use it 2024-01-21T19:44:51.573 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks 2024-01-21T19:44:51.573 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-21T19:44:51.574 INFO:teuthology.task.internal:Checking packages... 2024-01-21T19:44:51.596 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec' 2024-01-21T19:44:51.596 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-21T19:44:51.596 INFO:teuthology.packaging:ref: None 2024-01-21T19:44:51.596 INFO:teuthology.packaging:tag: None 2024-01-21T19:44:51.596 INFO:teuthology.packaging:branch: wip-yuri4-testing-2024-01-18-1257-pacific 2024-01-21T19:44:51.596 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T19:44:51.596 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri4-testing-2024-01-18-1257-pacific 2024-01-21T19:44:51.819 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-454.gea310589 2024-01-21T19:44:51.820 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-21T19:44:51.827 INFO:teuthology.task.internal:no buildpackages task found 2024-01-21T19:44:51.827 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-21T19:44:51.891 INFO:teuthology.task.internal:Saving configuration 2024-01-21T19:44:51.909 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-21T19:44:51.915 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-21T19:44:51.940 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi053.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-21_16:37:18-fs-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7525069', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-21 19:39:41.954777', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKpOsOjovXAabvq3jBF/C3fX8dBGkOjGUi3rKmXmWHvlzOiZgPGWaB05cvinr0wkjK+mL2LXbbaLTbta922SPeA='} 2024-01-21T19:44:51.961 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi146.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-21_16:37:18-fs-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7525069', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-21 19:39:41.956506', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD3QXMV9nD31+bSTFbZE+8Kvev917q1qJtUgAKnX7s/UtLVRVFfrd0QefK8rG/D6DvyvClaX7OrgeJnt77fF0UQ='} 2024-01-21T19:44:51.961 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-21T19:44:51.968 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-01-21T19:44:51.968 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-01-21T19:44:51.968 INFO:teuthology.run_tasks:Running task console_log... 2024-01-21T19:44:52.039 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f613f19c160>, signals=[15]) 2024-01-21T19:44:52.039 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-21T19:44:52.046 INFO:teuthology.task.internal:Opening connections... 2024-01-21T19:44:52.046 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi053.front.sepia.ceph.com 2024-01-21T19:44:52.047 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T19:44:52.121 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi146.front.sepia.ceph.com 2024-01-21T19:44:52.122 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T19:44:52.190 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-21T19:44:52.197 DEBUG:teuthology.orchestra.run.smithi053:> uname -m 2024-01-21T19:44:52.220 INFO:teuthology.orchestra.run.smithi053.stdout:x86_64 2024-01-21T19:44:52.221 DEBUG:teuthology.orchestra.run.smithi053:> cat /etc/os-release 2024-01-21T19:44:52.283 INFO:teuthology.orchestra.run.smithi053.stdout:NAME="CentOS Stream" 2024-01-21T19:44:52.283 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION="8" 2024-01-21T19:44:52.283 INFO:teuthology.orchestra.run.smithi053.stdout:ID="centos" 2024-01-21T19:44:52.284 INFO:teuthology.orchestra.run.smithi053.stdout:ID_LIKE="rhel fedora" 2024-01-21T19:44:52.284 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION_ID="8" 2024-01-21T19:44:52.284 INFO:teuthology.orchestra.run.smithi053.stdout:PLATFORM_ID="platform:el8" 2024-01-21T19:44:52.284 INFO:teuthology.orchestra.run.smithi053.stdout:PRETTY_NAME="CentOS Stream 8" 2024-01-21T19:44:52.284 INFO:teuthology.orchestra.run.smithi053.stdout:ANSI_COLOR="0;31" 2024-01-21T19:44:52.284 INFO:teuthology.orchestra.run.smithi053.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-01-21T19:44:52.284 INFO:teuthology.orchestra.run.smithi053.stdout:HOME_URL="https://centos.org/" 2024-01-21T19:44:52.284 INFO:teuthology.orchestra.run.smithi053.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-21T19:44:52.284 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-21T19:44:52.285 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-21T19:44:52.285 INFO:teuthology.lock.ops:Updating smithi053.front.sepia.ceph.com on lock server 2024-01-21T19:44:52.309 DEBUG:teuthology.orchestra.run.smithi146:> uname -m 2024-01-21T19:44:52.331 INFO:teuthology.orchestra.run.smithi146.stdout:x86_64 2024-01-21T19:44:52.331 DEBUG:teuthology.orchestra.run.smithi146:> cat /etc/os-release 2024-01-21T19:44:52.392 INFO:teuthology.orchestra.run.smithi146.stdout:NAME="CentOS Stream" 2024-01-21T19:44:52.392 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION="8" 2024-01-21T19:44:52.392 INFO:teuthology.orchestra.run.smithi146.stdout:ID="centos" 2024-01-21T19:44:52.392 INFO:teuthology.orchestra.run.smithi146.stdout:ID_LIKE="rhel fedora" 2024-01-21T19:44:52.393 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION_ID="8" 2024-01-21T19:44:52.393 INFO:teuthology.orchestra.run.smithi146.stdout:PLATFORM_ID="platform:el8" 2024-01-21T19:44:52.393 INFO:teuthology.orchestra.run.smithi146.stdout:PRETTY_NAME="CentOS Stream 8" 2024-01-21T19:44:52.393 INFO:teuthology.orchestra.run.smithi146.stdout:ANSI_COLOR="0;31" 2024-01-21T19:44:52.393 INFO:teuthology.orchestra.run.smithi146.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-01-21T19:44:52.393 INFO:teuthology.orchestra.run.smithi146.stdout:HOME_URL="https://centos.org/" 2024-01-21T19:44:52.393 INFO:teuthology.orchestra.run.smithi146.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-21T19:44:52.393 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-21T19:44:52.393 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-21T19:44:52.393 INFO:teuthology.lock.ops:Updating smithi146.front.sepia.ceph.com on lock server 2024-01-21T19:44:52.417 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-21T19:44:52.426 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-21T19:44:52.447 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-21T19:44:52.448 DEBUG:teuthology.orchestra.run.smithi053:> test '!' -e /home/ubuntu/cephtest 2024-01-21T19:44:52.450 DEBUG:teuthology.orchestra.run.smithi146:> test '!' -e /home/ubuntu/cephtest 2024-01-21T19:44:52.470 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-21T19:44:52.476 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-21T19:44:52.476 DEBUG:teuthology.orchestra.run.smithi053:> test -z $(ls -A /var/lib/ceph) 2024-01-21T19:44:52.511 DEBUG:teuthology.orchestra.run.smithi146:> test -z $(ls -A /var/lib/ceph) 2024-01-21T19:44:52.549 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-21T19:44:52.601 INFO:teuthology.run_tasks:Running task kernel... 2024-01-21T19:44:52.615 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-01-21T19:44:52.616 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-01-21T19:44:52.616 DEBUG:teuthology.orchestra.run.smithi053:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-21T19:44:52.616 DEBUG:teuthology.orchestra.run.smithi146:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-21T19:44:52.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T19:44:52.637 DEBUG:teuthology.orchestra.run.smithi146:> uname -r 2024-01-21T19:44:52.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T19:44:52.638 DEBUG:teuthology.orchestra.run.smithi053:> uname -r 2024-01-21T19:44:52.697 INFO:teuthology.orchestra.run.smithi146.stdout:4.18.0-489.el8.x86_64 2024-01-21T19:44:52.697 INFO:teuthology.task.kernel:Running kernel on smithi146: 4.18.0-489.el8.x86_64 2024-01-21T19:44:52.697 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum install -y kernel 2024-01-21T19:44:52.701 INFO:teuthology.orchestra.run.smithi053.stdout:4.18.0-489.el8.x86_64 2024-01-21T19:44:52.701 INFO:teuthology.task.kernel:Running kernel on smithi053: 4.18.0-489.el8.x86_64 2024-01-21T19:44:52.701 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum install -y kernel 2024-01-21T19:44:57.193 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 8 - AppStream 14 MB/s | 26 MB 00:01 2024-01-21T19:45:01.988 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 8 - AppStream 3.9 MB/s | 26 MB 00:06 2024-01-21T19:45:03.186 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 8 - BaseOS 9.9 MB/s | 10 MB 00:01 2024-01-21T19:45:07.136 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 8 - Extras 7.9 kB/s | 18 kB 00:02 2024-01-21T19:45:07.939 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 8 - BaseOS 8.5 MB/s | 10 MB 00:01 2024-01-21T19:45:08.020 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 8 - Extras common packages 9.1 kB/s | 6.9 kB 00:00 2024-01-21T19:45:08.346 INFO:teuthology.orchestra.run.smithi053.stdout:Copr repo for python3-asyncssh owned by ceph 15 kB/s | 3.5 kB 00:00 2024-01-21T19:45:10.489 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 8 - Extras 19 kB/s | 18 kB 00:00 2024-01-21T19:45:10.998 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 8 - Extras common packages 18 kB/s | 6.9 kB 00:00 2024-01-21T19:45:11.392 INFO:teuthology.orchestra.run.smithi146.stdout:Copr repo for python3-asyncssh owned by ceph 11 kB/s | 3.5 kB 00:00 2024-01-21T19:45:12.566 INFO:teuthology.orchestra.run.smithi053.stdout:Extra Packages for Enterprise Linux 3.9 MB/s | 16 MB 00:04 2024-01-21T19:45:12.911 INFO:teuthology.orchestra.run.smithi146.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2024-01-21T19:45:16.642 INFO:teuthology.orchestra.run.smithi146.stdout:lab-extras 631 kB/s | 24 kB 00:00 2024-01-21T19:45:16.655 INFO:teuthology.orchestra.run.smithi053.stdout:lab-extras 435 kB/s | 24 kB 00:00 2024-01-21T19:45:19.547 INFO:teuthology.orchestra.run.smithi146.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-01-21T19:45:19.547 INFO:teuthology.orchestra.run.smithi146.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-01-21T19:45:19.669 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-01-21T19:45:19.670 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:45:19.670 INFO:teuthology.orchestra.run.smithi146.stdout: Package Architecture Version Repository Size 2024-01-21T19:45:19.670 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:45:19.670 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-01-21T19:45:19.670 INFO:teuthology.orchestra.run.smithi146.stdout: kernel x86_64 4.18.0-535.el8 baseos 10 M 2024-01-21T19:45:19.670 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-01-21T19:45:19.671 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-core x86_64 4.18.0-535.el8 baseos 43 M 2024-01-21T19:45:19.671 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-modules x86_64 4.18.0-535.el8 baseos 36 M 2024-01-21T19:45:19.671 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:45:19.671 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-01-21T19:45:19.671 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:45:19.671 INFO:teuthology.orchestra.run.smithi146.stdout:Install 3 Packages 2024-01-21T19:45:19.671 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:45:19.671 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 89 M 2024-01-21T19:45:19.672 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 96 M 2024-01-21T19:45:19.672 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-01-21T19:45:19.796 INFO:teuthology.orchestra.run.smithi053.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-01-21T19:45:19.796 INFO:teuthology.orchestra.run.smithi053.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-01-21T19:45:19.897 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-01-21T19:45:19.898 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:45:19.898 INFO:teuthology.orchestra.run.smithi053.stdout: Package Architecture Version Repository Size 2024-01-21T19:45:19.898 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:45:19.898 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-01-21T19:45:19.898 INFO:teuthology.orchestra.run.smithi053.stdout: kernel x86_64 4.18.0-535.el8 baseos 10 M 2024-01-21T19:45:19.898 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-01-21T19:45:19.898 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-core x86_64 4.18.0-535.el8 baseos 43 M 2024-01-21T19:45:19.898 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-modules x86_64 4.18.0-535.el8 baseos 36 M 2024-01-21T19:45:19.898 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:45:19.898 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-01-21T19:45:19.898 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:45:19.899 INFO:teuthology.orchestra.run.smithi053.stdout:Install 3 Packages 2024-01-21T19:45:19.899 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:45:19.899 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 89 M 2024-01-21T19:45:19.899 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 96 M 2024-01-21T19:45:19.899 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-01-21T19:45:21.137 INFO:teuthology.orchestra.run.smithi146.stdout:(1/3): kernel-4.18.0-535.el8.x86_64.rpm 8.2 MB/s | 10 MB 00:01 2024-01-21T19:45:21.289 INFO:teuthology.orchestra.run.smithi053.stdout:(1/3): kernel-4.18.0-535.el8.x86_64.rpm 7.7 MB/s | 10 MB 00:01 2024-01-21T19:45:22.687 INFO:teuthology.orchestra.run.smithi146.stdout:(2/3): kernel-modules-4.18.0-535.el8.x86_64.rpm 13 MB/s | 36 MB 00:02 2024-01-21T19:45:22.931 INFO:teuthology.orchestra.run.smithi053.stdout:(2/3): kernel-modules-4.18.0-535.el8.x86_64.rpm 12 MB/s | 36 MB 00:02 2024-01-21T19:45:23.148 INFO:teuthology.orchestra.run.smithi146.stdout:(3/3): kernel-core-4.18.0-535.el8.x86_64.rpm 13 MB/s | 43 MB 00:03 2024-01-21T19:45:23.148 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:45:23.148 INFO:teuthology.orchestra.run.smithi146.stdout:Total 26 MB/s | 89 MB 00:03 2024-01-21T19:45:23.390 INFO:teuthology.orchestra.run.smithi053.stdout:(3/3): kernel-core-4.18.0-535.el8.x86_64.rpm 13 MB/s | 43 MB 00:03 2024-01-21T19:45:23.390 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:45:23.390 INFO:teuthology.orchestra.run.smithi053.stdout:Total 26 MB/s | 89 MB 00:03 2024-01-21T19:45:23.832 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-01-21T19:45:24.100 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-01-21T19:45:24.235 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-01-21T19:45:24.236 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-01-21T19:45:24.569 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-01-21T19:45:24.570 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-01-21T19:45:25.476 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-01-21T19:45:25.499 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-01-21T19:45:25.889 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-01-21T19:45:25.911 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-01-21T19:45:28.245 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-01-21T19:45:28.794 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : kernel-core-4.18.0-535.el8.x86_64 1/3 2024-01-21T19:45:28.834 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-01-21T19:45:29.301 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : kernel-core-4.18.0-535.el8.x86_64 1/3 2024-01-21T19:45:32.402 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-core-4.18.0-535.el8.x86_64 1/3 2024-01-21T19:45:32.818 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : kernel-modules-4.18.0-535.el8.x86_64 2/3 2024-01-21T19:45:33.091 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-core-4.18.0-535.el8.x86_64 1/3 2024-01-21T19:45:33.484 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : kernel-modules-4.18.0-535.el8.x86_64 2/3 2024-01-21T19:45:36.244 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-modules-4.18.0-535.el8.x86_64 2/3 2024-01-21T19:45:36.541 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : kernel-4.18.0-535.el8.x86_64 3/3 2024-01-21T19:45:37.159 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-modules-4.18.0-535.el8.x86_64 2/3 2024-01-21T19:45:37.461 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : kernel-4.18.0-535.el8.x86_64 3/3 2024-01-21T19:45:58.403 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-core-4.18.0-535.el8.x86_64 3/3 2024-01-21T19:45:58.426 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-modules-4.18.0-535.el8.x86_64 3/3 2024-01-21T19:45:58.779 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-4.18.0-535.el8.x86_64 3/3 2024-01-21T19:45:58.779 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : kernel-4.18.0-535.el8.x86_64 1/3 2024-01-21T19:45:58.779 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : kernel-core-4.18.0-535.el8.x86_64 2/3 2024-01-21T19:45:59.247 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : kernel-modules-4.18.0-535.el8.x86_64 3/3 2024-01-21T19:45:59.247 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:45:59.247 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-01-21T19:45:59.247 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-4.18.0-535.el8.x86_64 kernel-core-4.18.0-535.el8.x86_64 2024-01-21T19:45:59.247 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-modules-4.18.0-535.el8.x86_64 2024-01-21T19:45:59.247 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:45:59.248 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-01-21T19:45:59.395 DEBUG:teuthology.orchestra.run.smithi146:> rpm -q kernel | sort -rV | head -n 1 2024-01-21T19:45:59.636 INFO:teuthology.orchestra.run.smithi146.stdout:kernel-4.18.0-535.el8.x86_64 2024-01-21T19:45:59.636 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-535.el8.x86_64 2024-01-21T19:45:59.636 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-535.el8.x86_64 2024-01-21T19:45:59.636 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-21T19:45:59.636 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-21T19:45:59.637 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-01-21T19:45:59.637 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi146.front.sepia.ceph.com, path=None, version=distro) 2024-01-21T19:45:59.637 DEBUG:teuthology.orchestra.run.smithi146:> rpm -q kernel | sort -rV | head -n 1 2024-01-21T19:45:59.847 INFO:teuthology.orchestra.run.smithi146.stdout:kernel-4.18.0-535.el8.x86_64 2024-01-21T19:45:59.847 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-535.el8.x86_64 2024-01-21T19:45:59.847 DEBUG:teuthology.orchestra.run.smithi146:> sudo rpm -qi grub2-tools 2024-01-21T19:45:59.881 INFO:teuthology.orchestra.run.smithi146.stdout:Name : grub2-tools 2024-01-21T19:45:59.881 INFO:teuthology.orchestra.run.smithi146.stdout:Epoch : 1 2024-01-21T19:45:59.881 INFO:teuthology.orchestra.run.smithi146.stdout:Version : 2.02 2024-01-21T19:45:59.881 INFO:teuthology.orchestra.run.smithi146.stdout:Release : 129.el8 2024-01-21T19:45:59.881 INFO:teuthology.orchestra.run.smithi146.stdout:Architecture: x86_64 2024-01-21T19:45:59.881 INFO:teuthology.orchestra.run.smithi146.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-01-21T19:45:59.881 INFO:teuthology.orchestra.run.smithi146.stdout:Group : System Environment/Base 2024-01-21T19:45:59.881 INFO:teuthology.orchestra.run.smithi146.stdout:Size : 9265586 2024-01-21T19:45:59.882 INFO:teuthology.orchestra.run.smithi146.stdout:License : GPLv3+ 2024-01-21T19:45:59.882 INFO:teuthology.orchestra.run.smithi146.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-01-21T19:45:59.882 INFO:teuthology.orchestra.run.smithi146.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-01-21T19:45:59.882 INFO:teuthology.orchestra.run.smithi146.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-01-21T19:45:59.882 INFO:teuthology.orchestra.run.smithi146.stdout:Build Host : kbuilder.bsys.centos.org 2024-01-21T19:45:59.882 INFO:teuthology.orchestra.run.smithi146.stdout:Relocations : (not relocatable) 2024-01-21T19:45:59.882 INFO:teuthology.orchestra.run.smithi146.stdout:Packager : CentOS BuildSystem 2024-01-21T19:45:59.882 INFO:teuthology.orchestra.run.smithi146.stdout:Vendor : CentOS 2024-01-21T19:45:59.882 INFO:teuthology.orchestra.run.smithi146.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-21T19:45:59.882 INFO:teuthology.orchestra.run.smithi146.stdout:Summary : Support tools for GRUB. 2024-01-21T19:45:59.882 INFO:teuthology.orchestra.run.smithi146.stdout:Description : 2024-01-21T19:45:59.882 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:45:59.882 INFO:teuthology.orchestra.run.smithi146.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-21T19:45:59.883 INFO:teuthology.orchestra.run.smithi146.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-21T19:45:59.883 INFO:teuthology.orchestra.run.smithi146.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-21T19:45:59.883 INFO:teuthology.orchestra.run.smithi146.stdout:hardware devices. 2024-01-21T19:45:59.883 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:45:59.883 INFO:teuthology.orchestra.run.smithi146.stdout:This subpackage provides tools for support of all platforms. 2024-01-21T19:45:59.884 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-21T19:45:59.884 INFO:teuthology.task.kernel:Updating grub on smithi146 to boot 4.18.0-535.el8.x86_64 2024-01-21T19:45:59.884 DEBUG:teuthology.orchestra.run.smithi146:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-21T19:46:00.853 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-core-4.18.0-535.el8.x86_64 3/3 2024-01-21T19:46:00.876 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-modules-4.18.0-535.el8.x86_64 3/3 2024-01-21T19:46:01.252 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-4.18.0-535.el8.x86_64 3/3 2024-01-21T19:46:01.252 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : kernel-4.18.0-535.el8.x86_64 1/3 2024-01-21T19:46:01.252 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : kernel-core-4.18.0-535.el8.x86_64 2/3 2024-01-21T19:46:01.662 INFO:teuthology.orchestra.run.smithi146.stderr:Generating grub configuration file ... 2024-01-21T19:46:01.878 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : kernel-modules-4.18.0-535.el8.x86_64 3/3 2024-01-21T19:46:01.878 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:46:01.878 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-01-21T19:46:01.878 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-4.18.0-535.el8.x86_64 kernel-core-4.18.0-535.el8.x86_64 2024-01-21T19:46:01.878 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-modules-4.18.0-535.el8.x86_64 2024-01-21T19:46:01.878 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:46:01.878 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-01-21T19:46:02.028 DEBUG:teuthology.orchestra.run.smithi053:> rpm -q kernel | sort -rV | head -n 1 2024-01-21T19:46:02.307 INFO:teuthology.orchestra.run.smithi053.stdout:kernel-4.18.0-535.el8.x86_64 2024-01-21T19:46:02.307 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-535.el8.x86_64 2024-01-21T19:46:02.307 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-535.el8.x86_64 2024-01-21T19:46:02.307 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-21T19:46:02.308 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-21T19:46:02.308 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-01-21T19:46:02.308 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi053.front.sepia.ceph.com, path=None, version=distro) 2024-01-21T19:46:02.308 DEBUG:teuthology.orchestra.run.smithi053:> rpm -q kernel | sort -rV | head -n 1 2024-01-21T19:46:02.526 INFO:teuthology.orchestra.run.smithi053.stdout:kernel-4.18.0-535.el8.x86_64 2024-01-21T19:46:02.527 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-535.el8.x86_64 2024-01-21T19:46:02.527 DEBUG:teuthology.orchestra.run.smithi053:> sudo rpm -qi grub2-tools 2024-01-21T19:46:02.561 INFO:teuthology.orchestra.run.smithi053.stdout:Name : grub2-tools 2024-01-21T19:46:02.561 INFO:teuthology.orchestra.run.smithi053.stdout:Epoch : 1 2024-01-21T19:46:02.561 INFO:teuthology.orchestra.run.smithi053.stdout:Version : 2.02 2024-01-21T19:46:02.562 INFO:teuthology.orchestra.run.smithi053.stdout:Release : 129.el8 2024-01-21T19:46:02.562 INFO:teuthology.orchestra.run.smithi053.stdout:Architecture: x86_64 2024-01-21T19:46:02.562 INFO:teuthology.orchestra.run.smithi053.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-01-21T19:46:02.562 INFO:teuthology.orchestra.run.smithi053.stdout:Group : System Environment/Base 2024-01-21T19:46:02.562 INFO:teuthology.orchestra.run.smithi053.stdout:Size : 9265586 2024-01-21T19:46:02.562 INFO:teuthology.orchestra.run.smithi053.stdout:License : GPLv3+ 2024-01-21T19:46:02.562 INFO:teuthology.orchestra.run.smithi053.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-01-21T19:46:02.562 INFO:teuthology.orchestra.run.smithi053.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-01-21T19:46:02.562 INFO:teuthology.orchestra.run.smithi053.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-01-21T19:46:02.562 INFO:teuthology.orchestra.run.smithi053.stdout:Build Host : kbuilder.bsys.centos.org 2024-01-21T19:46:02.562 INFO:teuthology.orchestra.run.smithi053.stdout:Relocations : (not relocatable) 2024-01-21T19:46:02.562 INFO:teuthology.orchestra.run.smithi053.stdout:Packager : CentOS BuildSystem 2024-01-21T19:46:02.562 INFO:teuthology.orchestra.run.smithi053.stdout:Vendor : CentOS 2024-01-21T19:46:02.562 INFO:teuthology.orchestra.run.smithi053.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-21T19:46:02.562 INFO:teuthology.orchestra.run.smithi053.stdout:Summary : Support tools for GRUB. 2024-01-21T19:46:02.563 INFO:teuthology.orchestra.run.smithi053.stdout:Description : 2024-01-21T19:46:02.563 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:46:02.563 INFO:teuthology.orchestra.run.smithi053.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-21T19:46:02.563 INFO:teuthology.orchestra.run.smithi053.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-21T19:46:02.563 INFO:teuthology.orchestra.run.smithi053.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-21T19:46:02.563 INFO:teuthology.orchestra.run.smithi053.stdout:hardware devices. 2024-01-21T19:46:02.563 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:46:02.563 INFO:teuthology.orchestra.run.smithi053.stdout:This subpackage provides tools for support of all platforms. 2024-01-21T19:46:02.564 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-21T19:46:02.564 INFO:teuthology.task.kernel:Updating grub on smithi053 to boot 4.18.0-535.el8.x86_64 2024-01-21T19:46:02.564 DEBUG:teuthology.orchestra.run.smithi053:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-21T19:46:03.744 INFO:teuthology.orchestra.run.smithi146.stderr:done 2024-01-21T19:46:03.745 DEBUG:teuthology.orchestra.run.smithi146:> mktemp 2024-01-21T19:46:03.779 INFO:teuthology.orchestra.run.smithi146.stdout:/tmp/tmp.rwhk1thbKl 2024-01-21T19:46:03.779 DEBUG:teuthology.orchestra.run.smithi146:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.rwhk1thbKl 2024-01-21T19:46:03.853 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0666 /tmp/tmp.rwhk1thbKl 2024-01-21T19:46:03.997 DEBUG:teuthology.orchestra.remote:smithi146:/tmp/tmp.rwhk1thbKl is 6KB 2024-01-21T19:46:04.046 DEBUG:teuthology.orchestra.run.smithi146:> rm -fr /tmp/tmp.rwhk1thbKl 2024-01-21T19:46:04.064 DEBUG:teuthology.orchestra.run.smithi146:> sudo /bin/ls /boot/loader/entries || true 2024-01-21T19:46:04.125 INFO:teuthology.orchestra.run.smithi053.stderr:Generating grub configuration file ... 2024-01-21T19:46:04.142 INFO:teuthology.orchestra.run.smithi146.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-01-21T19:46:04.142 INFO:teuthology.orchestra.run.smithi146.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-01-21T19:46:04.142 INFO:teuthology.orchestra.run.smithi146.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-01-21T19:46:04.142 INFO:teuthology.orchestra.run.smithi146.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-535.el8.x86_64.conf 2024-01-21T19:46:04.143 DEBUG:teuthology.orchestra.run.smithi146:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-535.el8.x86_64 2024-01-21T19:46:04.312 DEBUG:teuthology.orchestra.run.smithi146:> sudo shutdown -r now 2024-01-21T19:46:06.211 INFO:teuthology.orchestra.run.smithi053.stderr:done 2024-01-21T19:46:06.212 DEBUG:teuthology.orchestra.run.smithi053:> mktemp 2024-01-21T19:46:06.238 INFO:teuthology.orchestra.run.smithi053.stdout:/tmp/tmp.0zCaCOFGDB 2024-01-21T19:46:06.239 DEBUG:teuthology.orchestra.run.smithi053:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.0zCaCOFGDB 2024-01-21T19:46:06.310 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 0666 /tmp/tmp.0zCaCOFGDB 2024-01-21T19:46:06.565 DEBUG:teuthology.orchestra.remote:smithi053:/tmp/tmp.0zCaCOFGDB is 6KB 2024-01-21T19:46:06.613 DEBUG:teuthology.orchestra.run.smithi053:> rm -fr /tmp/tmp.0zCaCOFGDB 2024-01-21T19:46:06.632 DEBUG:teuthology.orchestra.run.smithi053:> sudo /bin/ls /boot/loader/entries || true 2024-01-21T19:46:06.709 INFO:teuthology.orchestra.run.smithi053.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-01-21T19:46:06.709 INFO:teuthology.orchestra.run.smithi053.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-01-21T19:46:06.709 INFO:teuthology.orchestra.run.smithi053.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-01-21T19:46:06.709 INFO:teuthology.orchestra.run.smithi053.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-535.el8.x86_64.conf 2024-01-21T19:46:06.710 DEBUG:teuthology.orchestra.run.smithi053:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-535.el8.x86_64 2024-01-21T19:46:06.867 DEBUG:teuthology.orchestra.run.smithi053:> sudo shutdown -r now 2024-01-21T19:46:34.342 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-01-21T19:46:34.342 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-01-21T19:46:34.343 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T19:46:36.872 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-01-21T19:46:36.873 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-01-21T19:46:36.873 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T19:47:08.992 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-01-21T19:47:18.000 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-01-21T19:47:18.001 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T19:47:36.894 DEBUG:teuthology.orchestra.remote:timed out 2024-01-21T19:47:45.900 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-01-21T19:47:45.901 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T19:47:46.244 DEBUG:teuthology.orchestra.run.smithi053:> true 2024-01-21T19:47:46.520 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-01-21T19:47:46.520 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-535.el8.x86_64"... 2024-01-21T19:47:46.520 DEBUG:teuthology.orchestra.run.smithi053:> uname -r 2024-01-21T19:47:46.540 INFO:teuthology.orchestra.run.smithi053.stdout:4.18.0-535.el8.x86_64 2024-01-21T19:47:46.540 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-535.el8.x86_64 vs 4.18.0-535.el8.x86_64 2024-01-21T19:47:46.540 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-21T19:47:46.540 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-21T19:47:47.541 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-01-21T19:47:47.541 DEBUG:teuthology.orchestra.run.smithi053:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-21T19:47:47.634 INFO:teuthology.orchestra.run.smithi053.stdout:ttyS1 2024-01-21T19:47:47.650 DEBUG:teuthology.parallel:result is None 2024-01-21T19:48:18.003 DEBUG:teuthology.orchestra.remote:timed out 2024-01-21T19:48:30.003 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-01-21T19:48:30.005 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T19:48:30.324 DEBUG:teuthology.orchestra.run.smithi146:> true 2024-01-21T19:48:30.625 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-01-21T19:48:30.626 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-535.el8.x86_64"... 2024-01-21T19:48:30.626 DEBUG:teuthology.orchestra.run.smithi146:> uname -r 2024-01-21T19:48:30.645 INFO:teuthology.orchestra.run.smithi146.stdout:4.18.0-535.el8.x86_64 2024-01-21T19:48:30.645 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-535.el8.x86_64 vs 4.18.0-535.el8.x86_64 2024-01-21T19:48:30.645 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-21T19:48:30.646 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-21T19:48:31.647 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-01-21T19:48:31.647 DEBUG:teuthology.orchestra.run.smithi146:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-21T19:48:31.719 INFO:teuthology.orchestra.run.smithi146.stdout:ttyS1 2024-01-21T19:48:31.732 DEBUG:teuthology.parallel:result is None 2024-01-21T19:48:31.732 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-21T19:48:31.739 INFO:teuthology.task.internal:Creating test directory... 2024-01-21T19:48:31.739 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-21T19:48:31.742 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-21T19:48:31.767 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-21T19:48:31.774 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-21T19:48:31.781 INFO:teuthology.task.internal:Creating archive directory... 2024-01-21T19:48:31.781 DEBUG:teuthology.orchestra.run.smithi053:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-21T19:48:31.807 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-21T19:48:31.852 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-21T19:48:31.859 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-21T19:48:31.859 DEBUG:teuthology.orchestra.run.smithi053:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-21T19:48:31.887 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-21T19:48:31.921 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-21T19:48:31.925 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-21T19:48:31.933 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-21T19:48:31.936 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-21T19:48:31.937 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-21T19:48:31.961 INFO:teuthology.task.internal:Configuring sudo... 2024-01-21T19:48:31.961 DEBUG:teuthology.orchestra.run.smithi053:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-21T19:48:31.978 DEBUG:teuthology.orchestra.run.smithi146:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-21T19:48:32.008 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-21T19:48:32.018 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-21T19:48:32.018 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-21T19:48:32.049 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-21T19:48:32.068 DEBUG:teuthology.orchestra.run.smithi053:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-21T19:48:32.155 DEBUG:teuthology.orchestra.run.smithi053:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-21T19:48:32.210 DEBUG:teuthology.orchestra.run.smithi053:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-21T19:48:32.295 DEBUG:teuthology.orchestra.run.smithi053:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-21T19:48:32.342 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-21T19:48:32.342 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-21T19:48:32.412 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-21T19:48:32.459 DEBUG:teuthology.orchestra.run.smithi146:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-21T19:48:32.513 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-21T19:48:32.598 DEBUG:teuthology.orchestra.run.smithi146:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-21T19:48:32.642 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-21T19:48:32.642 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-21T19:48:32.710 DEBUG:teuthology.orchestra.run.smithi053:> sudo service rsyslog restart 2024-01-21T19:48:32.713 DEBUG:teuthology.orchestra.run.smithi146:> sudo service rsyslog restart 2024-01-21T19:48:32.792 INFO:teuthology.orchestra.run.smithi053.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-21T19:48:32.809 INFO:teuthology.orchestra.run.smithi146.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-21T19:48:33.261 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-21T19:48:33.269 INFO:teuthology.task.internal:Starting timer... 2024-01-21T19:48:33.269 INFO:teuthology.run_tasks:Running task pcp... 2024-01-21T19:48:33.279 INFO:teuthology.run_tasks:Running task selinux... 2024-01-21T19:48:33.287 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-01-21T19:48:33.287 DEBUG:teuthology.orchestra.run.smithi053:> sudo service auditd rotate 2024-01-21T19:48:33.373 INFO:teuthology.orchestra.run.smithi053.stdout:Rotating logs: 2024-01-21T19:48:33.375 DEBUG:teuthology.orchestra.run.smithi146:> sudo service auditd rotate 2024-01-21T19:48:33.450 INFO:teuthology.orchestra.run.smithi146.stdout:Rotating logs: 2024-01-21T19:48:33.452 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-21T19:48:33.452 DEBUG:teuthology.orchestra.run.smithi053:> /usr/sbin/getenforce 2024-01-21T19:48:33.504 INFO:teuthology.orchestra.run.smithi053.stdout:Permissive 2024-01-21T19:48:33.504 DEBUG:teuthology.orchestra.run.smithi146:> /usr/sbin/getenforce 2024-01-21T19:48:33.577 INFO:teuthology.orchestra.run.smithi146.stdout:Permissive 2024-01-21T19:48:33.577 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi053.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi146.front.sepia.ceph.com': 'permissive'} 2024-01-21T19:48:33.577 DEBUG:teuthology.orchestra.run.smithi053:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-21T19:48:33.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T19:48:33.607 DEBUG:teuthology.orchestra.run.smithi146:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-21T19:48:33.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T19:48:33.635 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-21T19:48:33.635 DEBUG:teuthology.orchestra.run.smithi053:> sudo /usr/sbin/setenforce permissive 2024-01-21T19:48:33.676 DEBUG:teuthology.orchestra.run.smithi146:> sudo /usr/sbin/setenforce permissive 2024-01-21T19:48:33.703 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-21T19:48:33.713 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-21T19:48:33.730 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-21T19:48:33.731 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi053.front.sepia.ceph.com,smithi146.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-21T19:54:18.011 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi053.front.sepia.ceph.com'), Remote(name='ubuntu@smithi146.front.sepia.ceph.com')] 2024-01-21T19:54:18.013 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-01-21T19:54:18.014 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T19:54:18.089 DEBUG:teuthology.orchestra.run.smithi053:> true 2024-01-21T19:54:18.164 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-01-21T19:54:18.164 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-01-21T19:54:18.165 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T19:54:18.238 DEBUG:teuthology.orchestra.run.smithi146:> true 2024-01-21T19:54:18.308 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-01-21T19:54:18.308 INFO:teuthology.run_tasks:Running task clock... 2024-01-21T19:54:18.319 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-21T19:54:18.319 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-21T19:54:18.319 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-21T19:54:18.322 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-21T19:54:18.322 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-21T19:54:18.356 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-21T19:54:18.372 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-21T19:54:18.382 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-21T19:54:18.396 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-21T19:54:18.447 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: ntpd: command not found 2024-01-21T19:54:18.458 INFO:teuthology.orchestra.run.smithi053.stdout:506 Cannot talk to daemon 2024-01-21T19:54:18.462 INFO:teuthology.orchestra.run.smithi146.stderr:sudo: ntpd: command not found 2024-01-21T19:54:18.473 INFO:teuthology.orchestra.run.smithi146.stdout:506 Cannot talk to daemon 2024-01-21T19:54:18.473 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-21T19:54:18.487 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-21T19:54:18.487 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-21T19:54:18.500 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-21T19:54:18.557 INFO:teuthology.orchestra.run.smithi053.stderr:bash: ntpq: command not found 2024-01-21T19:54:18.561 INFO:teuthology.orchestra.run.smithi053.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-21T19:54:18.561 INFO:teuthology.orchestra.run.smithi053.stdout:=============================================================================== 2024-01-21T19:54:18.561 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T19:54:18.561 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T19:54:18.561 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T19:54:18.561 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T19:54:18.573 INFO:teuthology.orchestra.run.smithi146.stderr:bash: ntpq: command not found 2024-01-21T19:54:18.576 INFO:teuthology.orchestra.run.smithi146.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-21T19:54:18.576 INFO:teuthology.orchestra.run.smithi146.stdout:=============================================================================== 2024-01-21T19:54:18.576 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T19:54:18.576 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T19:54:18.576 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T19:54:18.576 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T19:54:18.577 INFO:teuthology.run_tasks:Running task pexec... 2024-01-21T19:54:18.587 INFO:teuthology.task.pexec:Executing custom commands... 2024-01-21T19:54:18.587 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi053.front.sepia.ceph.com 2024-01-21T19:54:18.587 DEBUG:teuthology.orchestra.run.smithi053:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-21T19:54:18.588 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi146.front.sepia.ceph.com 2024-01-21T19:54:18.588 DEBUG:teuthology.orchestra.run.smithi146:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-21T19:54:19.375 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:28 ago on Sun 21 Jan 2024 07:52:51 PM UTC. 2024-01-21T19:54:19.389 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:21 ago on Sun 21 Jan 2024 07:52:58 PM UTC. 2024-01-21T19:54:21.293 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-01-21T19:54:21.294 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:54:21.295 INFO:teuthology.orchestra.run.smithi146.stdout: Package Architecture Version Repository Size 2024-01-21T19:54:21.295 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:54:21.295 INFO:teuthology.orchestra.run.smithi146.stdout:Resetting modules: 2024-01-21T19:54:21.295 INFO:teuthology.orchestra.run.smithi146.stdout: container-tools 2024-01-21T19:54:21.295 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:54:21.295 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-01-21T19:54:21.295 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:54:21.295 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:54:21.448 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-01-21T19:54:21.449 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:54:21.449 INFO:teuthology.orchestra.run.smithi053.stdout: Package Architecture Version Repository Size 2024-01-21T19:54:21.449 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:54:21.449 INFO:teuthology.orchestra.run.smithi053.stdout:Resetting modules: 2024-01-21T19:54:21.449 INFO:teuthology.orchestra.run.smithi053.stdout: container-tools 2024-01-21T19:54:21.449 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:54:21.449 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-01-21T19:54:21.450 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:54:21.450 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:54:21.544 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-01-21T19:54:21.726 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-01-21T19:54:22.404 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:31 ago on Sun 21 Jan 2024 07:52:51 PM UTC. 2024-01-21T19:54:22.614 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:24 ago on Sun 21 Jan 2024 07:52:58 PM UTC. 2024-01-21T19:54:24.359 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-01-21T19:54:24.393 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:54:24.394 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-01-21T19:54:24.394 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:54:24.394 INFO:teuthology.orchestra.run.smithi146.stdout:Installing group/module packages: 2024-01-21T19:54:24.394 INFO:teuthology.orchestra.run.smithi146.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2024-01-21T19:54:24.394 INFO:teuthology.orchestra.run.smithi146.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2024-01-21T19:54:24.394 INFO:teuthology.orchestra.run.smithi146.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2024-01-21T19:54:24.394 INFO:teuthology.orchestra.run.smithi146.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2024-01-21T19:54:24.394 INFO:teuthology.orchestra.run.smithi146.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2024-01-21T19:54:24.394 INFO:teuthology.orchestra.run.smithi146.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2024-01-21T19:54:24.394 INFO:teuthology.orchestra.run.smithi146.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2024-01-21T19:54:24.394 INFO:teuthology.orchestra.run.smithi146.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2024-01-21T19:54:24.394 INFO:teuthology.orchestra.run.smithi146.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2024-01-21T19:54:24.395 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-01-21T19:54:24.395 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-01-21T19:54:24.395 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-01-21T19:54:24.395 INFO:teuthology.orchestra.run.smithi146.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-01-21T19:54:24.395 INFO:teuthology.orchestra.run.smithi146.stdout:Installing weak dependencies: 2024-01-21T19:54:24.395 INFO:teuthology.orchestra.run.smithi146.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-01-21T19:54:24.395 INFO:teuthology.orchestra.run.smithi146.stdout:Installing module profiles: 2024-01-21T19:54:24.395 INFO:teuthology.orchestra.run.smithi146.stdout: container-tools/common 2024-01-21T19:54:24.395 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:54:24.395 INFO:teuthology.orchestra.run.smithi146.stdout:Enabling module streams: 2024-01-21T19:54:24.395 INFO:teuthology.orchestra.run.smithi146.stdout: container-tools rhel8 2024-01-21T19:54:24.395 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:54:24.395 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-01-21T19:54:24.395 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:54:24.395 INFO:teuthology.orchestra.run.smithi146.stdout:Install 13 Packages 2024-01-21T19:54:24.396 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:54:24.396 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 27 M 2024-01-21T19:54:24.396 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 89 M 2024-01-21T19:54:24.396 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-01-21T19:54:24.699 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-01-21T19:54:24.725 INFO:teuthology.orchestra.run.smithi146.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 2.6 MB/s | 750 kB 00:00 2024-01-21T19:54:24.737 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:54:24.737 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-01-21T19:54:24.737 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:54:24.738 INFO:teuthology.orchestra.run.smithi053.stdout:Installing group/module packages: 2024-01-21T19:54:24.738 INFO:teuthology.orchestra.run.smithi053.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2024-01-21T19:54:24.738 INFO:teuthology.orchestra.run.smithi053.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2024-01-21T19:54:24.738 INFO:teuthology.orchestra.run.smithi053.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2024-01-21T19:54:24.738 INFO:teuthology.orchestra.run.smithi053.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2024-01-21T19:54:24.738 INFO:teuthology.orchestra.run.smithi053.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2024-01-21T19:54:24.738 INFO:teuthology.orchestra.run.smithi053.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2024-01-21T19:54:24.738 INFO:teuthology.orchestra.run.smithi053.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2024-01-21T19:54:24.738 INFO:teuthology.orchestra.run.smithi053.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2024-01-21T19:54:24.738 INFO:teuthology.orchestra.run.smithi053.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2024-01-21T19:54:24.738 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-01-21T19:54:24.738 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-01-21T19:54:24.738 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-01-21T19:54:24.739 INFO:teuthology.orchestra.run.smithi053.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-01-21T19:54:24.739 INFO:teuthology.orchestra.run.smithi053.stdout:Installing weak dependencies: 2024-01-21T19:54:24.739 INFO:teuthology.orchestra.run.smithi053.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-01-21T19:54:24.739 INFO:teuthology.orchestra.run.smithi053.stdout:Installing module profiles: 2024-01-21T19:54:24.739 INFO:teuthology.orchestra.run.smithi053.stdout: container-tools/common 2024-01-21T19:54:24.739 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:54:24.739 INFO:teuthology.orchestra.run.smithi053.stdout:Enabling module streams: 2024-01-21T19:54:24.739 INFO:teuthology.orchestra.run.smithi053.stdout: container-tools rhel8 2024-01-21T19:54:24.739 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:54:24.739 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-01-21T19:54:24.739 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:54:24.739 INFO:teuthology.orchestra.run.smithi053.stdout:Install 13 Packages 2024-01-21T19:54:24.740 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:54:24.740 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 27 M 2024-01-21T19:54:24.740 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 89 M 2024-01-21T19:54:24.741 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-01-21T19:54:24.775 INFO:teuthology.orchestra.run.smithi146.stdout:(2/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 3.0 MB/s | 1.0 MB 00:00 2024-01-21T19:54:24.801 INFO:teuthology.orchestra.run.smithi146.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 503 kB/s | 38 kB 00:00 2024-01-21T19:54:24.859 INFO:teuthology.orchestra.run.smithi146.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 2.8 MB/s | 241 kB 00:00 2024-01-21T19:54:24.926 INFO:teuthology.orchestra.run.smithi146.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 2.6 MB/s | 178 kB 00:00 2024-01-21T19:54:25.001 INFO:teuthology.orchestra.run.smithi146.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 330 kB/s | 25 kB 00:00 2024-01-21T19:54:25.056 INFO:teuthology.orchestra.run.smithi053.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 2.7 MB/s | 750 kB 00:00 2024-01-21T19:54:25.068 INFO:teuthology.orchestra.run.smithi146.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.4 MB/s | 94 kB 00:00 2024-01-21T19:54:25.125 INFO:teuthology.orchestra.run.smithi053.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 553 kB/s | 38 kB 00:00 2024-01-21T19:54:25.165 INFO:teuthology.orchestra.run.smithi053.stdout:(3/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 2.6 MB/s | 1.0 MB 00:00 2024-01-21T19:54:25.190 INFO:teuthology.orchestra.run.smithi053.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 3.6 MB/s | 241 kB 00:00 2024-01-21T19:54:25.235 INFO:teuthology.orchestra.run.smithi146.stdout:(8/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 9.0 MB/s | 3.9 MB 00:00 2024-01-21T19:54:25.249 INFO:teuthology.orchestra.run.smithi053.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 3.0 MB/s | 178 kB 00:00 2024-01-21T19:54:25.291 INFO:teuthology.orchestra.run.smithi053.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 594 kB/s | 25 kB 00:00 2024-01-21T19:54:25.349 INFO:teuthology.orchestra.run.smithi053.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.6 MB/s | 94 kB 00:00 2024-01-21T19:54:25.527 INFO:teuthology.orchestra.run.smithi146.stdout:(9/13): buildah-1.32.0-1.module_el8+712+4cd1bd6 8.8 MB/s | 9.6 MB 00:01 2024-01-21T19:54:25.569 INFO:teuthology.orchestra.run.smithi146.stdout:(10/13): toolbox-0.0.99.4-6.module_el8+712+4cd1 7.6 MB/s | 2.5 MB 00:00 2024-01-21T19:54:25.594 INFO:teuthology.orchestra.run.smithi146.stdout:(11/13): udica-0.2.6-20.module_el8+712+4cd1bd69 786 kB/s | 53 kB 00:00 2024-01-21T19:54:25.628 INFO:teuthology.orchestra.run.smithi146.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 699 kB/s | 41 kB 00:00 2024-01-21T19:54:25.791 INFO:teuthology.orchestra.run.smithi053.stdout:(8/13): buildah-1.32.0-1.module_el8+712+4cd1bd6 9.5 MB/s | 9.6 MB 00:01 2024-01-21T19:54:25.828 INFO:teuthology.orchestra.run.smithi146.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 11 MB/s | 8.1 MB 00:00 2024-01-21T19:54:25.829 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:54:25.829 INFO:teuthology.orchestra.run.smithi146.stdout:Total 19 MB/s | 27 MB 00:01 2024-01-21T19:54:25.883 INFO:teuthology.orchestra.run.smithi053.stdout:(9/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 5.5 MB/s | 3.9 MB 00:00 2024-01-21T19:54:25.976 INFO:teuthology.orchestra.run.smithi053.stdout:(10/13): udica-0.2.6-20.module_el8+712+4cd1bd69 573 kB/s | 53 kB 00:00 2024-01-21T19:54:26.084 INFO:teuthology.orchestra.run.smithi053.stdout:(11/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 11 MB/s | 8.1 MB 00:00 2024-01-21T19:54:26.094 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-01-21T19:54:26.109 INFO:teuthology.orchestra.run.smithi053.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 307 kB/s | 41 kB 00:00 2024-01-21T19:54:26.115 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-01-21T19:54:26.115 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-01-21T19:54:26.160 INFO:teuthology.orchestra.run.smithi053.stdout:(13/13): toolbox-0.0.99.4-6.module_el8+712+4cd1 6.9 MB/s | 2.5 MB 00:00 2024-01-21T19:54:26.160 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:54:26.160 INFO:teuthology.orchestra.run.smithi053.stdout:Total 19 MB/s | 27 MB 00:01 2024-01-21T19:54:26.381 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-01-21T19:54:26.384 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-01-21T19:54:26.430 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-01-21T19:54:26.453 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-01-21T19:54:26.454 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-01-21T19:54:26.725 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-01-21T19:54:26.728 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-01-21T19:54:27.101 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-01-21T19:54:27.304 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-01-21T19:54:27.472 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-01-21T19:54:27.483 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-01-21T19:54:27.706 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2024-01-21T19:54:27.720 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-01-21T19:54:27.881 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-01-21T19:54:28.127 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2024-01-21T19:54:28.294 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-21T19:54:28.388 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-21T19:54:28.859 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2024-01-21T19:54:28.960 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2024-01-21T19:54:29.882 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-21T19:54:30.024 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-21T19:54:30.120 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2024-01-21T19:54:30.254 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2024-01-21T19:54:30.352 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2024-01-21T19:54:30.443 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2024-01-21T19:54:30.620 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2024-01-21T19:54:30.711 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2024-01-21T19:54:31.028 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2024-01-21T19:54:31.156 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2024-01-21T19:54:31.919 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-21T19:54:32.076 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-21T19:54:32.187 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2024-01-21T19:54:32.321 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2024-01-21T19:54:32.390 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-21T19:54:32.547 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-21T19:54:32.818 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-21T19:54:32.818 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2024-01-21T19:54:32.818 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2024-01-21T19:54:32.818 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2024-01-21T19:54:32.818 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-21T19:54:32.818 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2024-01-21T19:54:32.818 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-21T19:54:32.818 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2024-01-21T19:54:32.818 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-01-21T19:54:32.818 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-01-21T19:54:32.818 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2024-01-21T19:54:32.818 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-21T19:54:32.818 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2024-01-21T19:54:32.986 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-21T19:54:32.986 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2024-01-21T19:54:32.986 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2024-01-21T19:54:32.986 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2024-01-21T19:54:32.986 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-21T19:54:32.987 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2024-01-21T19:54:32.987 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-21T19:54:32.987 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2024-01-21T19:54:32.987 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-01-21T19:54:32.987 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-01-21T19:54:32.987 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2024-01-21T19:54:32.987 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-21T19:54:32.987 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2024-01-21T19:54:33.359 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-01-21T19:54:33.359 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:54:33.359 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-01-21T19:54:33.360 INFO:teuthology.orchestra.run.smithi146.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T19:54:33.360 INFO:teuthology.orchestra.run.smithi146.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T19:54:33.360 INFO:teuthology.orchestra.run.smithi146.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2024-01-21T19:54:33.360 INFO:teuthology.orchestra.run.smithi146.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-01-21T19:54:33.360 INFO:teuthology.orchestra.run.smithi146.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T19:54:33.360 INFO:teuthology.orchestra.run.smithi146.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2024-01-21T19:54:33.360 INFO:teuthology.orchestra.run.smithi146.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2024-01-21T19:54:33.360 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-01-21T19:54:33.360 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-01-21T19:54:33.360 INFO:teuthology.orchestra.run.smithi146.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T19:54:33.360 INFO:teuthology.orchestra.run.smithi146.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2024-01-21T19:54:33.360 INFO:teuthology.orchestra.run.smithi146.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2024-01-21T19:54:33.360 INFO:teuthology.orchestra.run.smithi146.stdout: yajl-2.1.0-12.el8.x86_64 2024-01-21T19:54:33.360 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:54:33.361 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-01-21T19:54:33.571 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-01-21T19:54:33.571 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:54:33.571 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-01-21T19:54:33.571 INFO:teuthology.orchestra.run.smithi053.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T19:54:33.571 INFO:teuthology.orchestra.run.smithi053.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T19:54:33.571 INFO:teuthology.orchestra.run.smithi053.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2024-01-21T19:54:33.571 INFO:teuthology.orchestra.run.smithi053.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-01-21T19:54:33.572 INFO:teuthology.orchestra.run.smithi053.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T19:54:33.572 INFO:teuthology.orchestra.run.smithi053.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2024-01-21T19:54:33.572 INFO:teuthology.orchestra.run.smithi053.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2024-01-21T19:54:33.572 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-01-21T19:54:33.572 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-01-21T19:54:33.572 INFO:teuthology.orchestra.run.smithi053.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T19:54:33.572 INFO:teuthology.orchestra.run.smithi053.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2024-01-21T19:54:33.572 INFO:teuthology.orchestra.run.smithi053.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2024-01-21T19:54:33.572 INFO:teuthology.orchestra.run.smithi053.stdout: yajl-2.1.0-12.el8.x86_64 2024-01-21T19:54:33.572 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:54:33.572 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-01-21T19:54:33.789 DEBUG:teuthology.parallel:result is None 2024-01-21T19:54:33.889 DEBUG:teuthology.parallel:result is None 2024-01-21T19:54:33.889 INFO:teuthology.run_tasks:Running task pexec... 2024-01-21T19:54:33.897 INFO:teuthology.task.pexec:Executing custom commands... 2024-01-21T19:54:33.898 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi053.front.sepia.ceph.com 2024-01-21T19:54:33.898 DEBUG:teuthology.orchestra.run.smithi053:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-21T19:54:33.898 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi146.front.sepia.ceph.com 2024-01-21T19:54:33.898 DEBUG:teuthology.orchestra.run.smithi146:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-21T19:54:34.122 DEBUG:teuthology.parallel:result is None 2024-01-21T19:54:34.156 DEBUG:teuthology.parallel:result is None 2024-01-21T19:54:34.156 INFO:teuthology.run_tasks:Running task install... 2024-01-21T19:54:34.164 DEBUG:teuthology.task.install:project ceph 2024-01-21T19:54:34.165 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec'}} 2024-01-21T19:54:34.165 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'flavor': 'default', 'sha1': 'ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec'} 2024-01-21T19:54:34.165 INFO:teuthology.task.install:Using flavor: default 2024-01-21T19:54:34.173 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-01-21T19:54:34.173 INFO:teuthology.task.install:extra packages: [] 2024-01-21T19:54:34.173 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-01-21T19:54:34.173 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-21T19:54:34.173 INFO:teuthology.packaging:ref: None 2024-01-21T19:54:34.173 INFO:teuthology.packaging:tag: v16.2.4 2024-01-21T19:54:34.173 INFO:teuthology.packaging:branch: None 2024-01-21T19:54:34.173 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T19:54:34.473 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-21T19:54:34.474 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-21T19:54:34.476 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-01-21T19:54:34.476 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-21T19:54:34.477 INFO:teuthology.packaging:ref: None 2024-01-21T19:54:34.477 INFO:teuthology.packaging:tag: v16.2.4 2024-01-21T19:54:34.477 INFO:teuthology.packaging:branch: None 2024-01-21T19:54:34.477 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T19:54:34.477 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-21T19:54:34.708 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-01-21T19:54:34.709 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-01-21T19:54:34.723 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-01-21T19:54:34.723 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-01-21T19:54:34.777 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-21T19:54:34.777 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-21T19:54:34.777 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-21T19:54:34.785 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-21T19:54:34.785 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-21T19:54:34.785 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-21T19:54:34.811 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-01-21T19:54:34.811 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-21T19:54:34.811 INFO:teuthology.packaging:ref: None 2024-01-21T19:54:34.811 INFO:teuthology.packaging:tag: v16.2.4 2024-01-21T19:54:34.811 INFO:teuthology.packaging:branch: None 2024-01-21T19:54:34.811 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T19:54:34.811 DEBUG:teuthology.orchestra.run.smithi053:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-21T19:54:34.814 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-01-21T19:54:34.814 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-21T19:54:34.814 INFO:teuthology.packaging:ref: None 2024-01-21T19:54:34.815 INFO:teuthology.packaging:tag: v16.2.4 2024-01-21T19:54:34.815 INFO:teuthology.packaging:branch: None 2024-01-21T19:54:34.815 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T19:54:34.815 DEBUG:teuthology.orchestra.run.smithi146:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-21T19:54:34.884 DEBUG:teuthology.orchestra.run.smithi146:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-21T19:54:34.885 DEBUG:teuthology.orchestra.run.smithi053:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-21T19:54:34.964 DEBUG:teuthology.orchestra.run.smithi146:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-21T19:54:34.967 DEBUG:teuthology.orchestra.run.smithi053:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-21T19:54:35.031 INFO:teuthology.orchestra.run.smithi146.stdout:check_obsoletes = 1 2024-01-21T19:54:35.032 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum clean all 2024-01-21T19:54:35.035 INFO:teuthology.orchestra.run.smithi053.stdout:check_obsoletes = 1 2024-01-21T19:54:35.037 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum clean all 2024-01-21T19:54:35.298 INFO:teuthology.orchestra.run.smithi146.stdout:76 files removed 2024-01-21T19:54:35.311 INFO:teuthology.orchestra.run.smithi053.stdout:76 files removed 2024-01-21T19:54:35.326 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-radosgw 2024-01-21T19:54:35.339 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-radosgw 2024-01-21T19:54:35.820 INFO:teuthology.orchestra.run.smithi146.stdout:ceph packages for x86_64 896 kB/s | 74 kB 00:00 2024-01-21T19:54:35.871 INFO:teuthology.orchestra.run.smithi053.stdout:ceph packages for x86_64 792 kB/s | 74 kB 00:00 2024-01-21T19:54:35.928 INFO:teuthology.orchestra.run.smithi146.stdout:ceph noarch packages 216 kB/s | 15 kB 00:00 2024-01-21T19:54:36.013 INFO:teuthology.orchestra.run.smithi053.stdout:ceph noarch packages 150 kB/s | 15 kB 00:00 2024-01-21T19:54:36.037 INFO:teuthology.orchestra.run.smithi146.stdout:ceph source packages 22 kB/s | 1.7 kB 00:00 2024-01-21T19:54:36.114 INFO:teuthology.orchestra.run.smithi053.stdout:ceph source packages 25 kB/s | 1.7 kB 00:00 2024-01-21T19:54:37.588 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 8 - AppStream 17 MB/s | 26 MB 00:01 2024-01-21T19:54:37.839 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 8 - AppStream 16 MB/s | 26 MB 00:01 2024-01-21T19:54:43.156 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 8 - BaseOS 13 MB/s | 10 MB 00:00 2024-01-21T19:54:43.557 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 8 - BaseOS 15 MB/s | 10 MB 00:00 2024-01-21T19:54:45.273 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 8 - Extras 33 kB/s | 18 kB 00:00 2024-01-21T19:54:45.740 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 8 - Extras common packages 20 kB/s | 6.9 kB 00:00 2024-01-21T19:54:45.817 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 8 - Extras 32 kB/s | 18 kB 00:00 2024-01-21T19:54:46.300 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 8 - Extras common packages 19 kB/s | 6.9 kB 00:00 2024-01-21T19:54:46.916 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 8 - PowerTools 4.4 MB/s | 4.7 MB 00:01 2024-01-21T19:54:47.537 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 8 - PowerTools 4.2 MB/s | 4.7 MB 00:01 2024-01-21T19:54:47.968 INFO:teuthology.orchestra.run.smithi146.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2024-01-21T19:54:48.703 INFO:teuthology.orchestra.run.smithi053.stdout:Copr repo for python3-asyncssh owned by ceph 15 kB/s | 3.5 kB 00:00 2024-01-21T19:54:49.721 INFO:teuthology.orchestra.run.smithi146.stdout:Extra Packages for Enterprise Linux 9.7 MB/s | 16 MB 00:01 2024-01-21T19:54:50.548 INFO:teuthology.orchestra.run.smithi053.stdout:Extra Packages for Enterprise Linux 9.2 MB/s | 16 MB 00:01 2024-01-21T19:54:53.527 INFO:teuthology.orchestra.run.smithi146.stdout:lab-extras 333 kB/s | 24 kB 00:00 2024-01-21T19:54:54.546 INFO:teuthology.orchestra.run.smithi053.stdout:lab-extras 409 kB/s | 24 kB 00:00 2024-01-21T19:54:57.032 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-01-21T19:54:57.035 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:54:57.035 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-01-21T19:54:57.035 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:54:57.035 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-01-21T19:54:57.035 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-01-21T19:54:57.035 INFO:teuthology.orchestra.run.smithi146.stdout:Upgrading: 2024-01-21T19:54:57.035 INFO:teuthology.orchestra.run.smithi146.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-21T19:54:57.035 INFO:teuthology.orchestra.run.smithi146.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-01-21T19:54:57.035 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-01-21T19:54:57.035 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-01-21T19:54:57.035 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-01-21T19:54:57.035 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-01-21T19:54:57.035 INFO:teuthology.orchestra.run.smithi146.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-01-21T19:54:57.036 INFO:teuthology.orchestra.run.smithi146.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-01-21T19:54:57.036 INFO:teuthology.orchestra.run.smithi146.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-01-21T19:54:57.036 INFO:teuthology.orchestra.run.smithi146.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-01-21T19:54:57.036 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-01-21T19:54:57.036 INFO:teuthology.orchestra.run.smithi146.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-01-21T19:54:57.036 INFO:teuthology.orchestra.run.smithi146.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-01-21T19:54:57.036 INFO:teuthology.orchestra.run.smithi146.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-01-21T19:54:57.036 INFO:teuthology.orchestra.run.smithi146.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-01-21T19:54:57.036 INFO:teuthology.orchestra.run.smithi146.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-01-21T19:54:57.036 INFO:teuthology.orchestra.run.smithi146.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-01-21T19:54:57.036 INFO:teuthology.orchestra.run.smithi146.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-01-21T19:54:57.036 INFO:teuthology.orchestra.run.smithi146.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-01-21T19:54:57.036 INFO:teuthology.orchestra.run.smithi146.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-01-21T19:54:57.036 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-01-21T19:54:57.036 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-01-21T19:54:57.037 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-01-21T19:54:57.037 INFO:teuthology.orchestra.run.smithi146.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-01-21T19:54:57.037 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-01-21T19:54:57.037 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-01-21T19:54:57.037 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-01-21T19:54:57.037 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:54:57.037 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-01-21T19:54:57.037 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:54:57.037 INFO:teuthology.orchestra.run.smithi146.stdout:Install 25 Packages 2024-01-21T19:54:57.037 INFO:teuthology.orchestra.run.smithi146.stdout:Upgrade 2 Packages 2024-01-21T19:54:57.037 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:54:57.038 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 59 M 2024-01-21T19:54:57.038 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-01-21T19:54:57.478 INFO:teuthology.orchestra.run.smithi146.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 40 MB/s | 6.0 MB 00:00 2024-01-21T19:54:57.503 INFO:teuthology.orchestra.run.smithi146.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 1.0 MB/s | 25 kB 00:00 2024-01-21T19:54:57.545 INFO:teuthology.orchestra.run.smithi146.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 19 MB/s | 801 kB 00:00 2024-01-21T19:54:57.579 INFO:teuthology.orchestra.run.smithi146.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2024-01-21T19:54:57.683 INFO:teuthology.orchestra.run.smithi146.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 39 MB/s | 4.1 MB 00:00 2024-01-21T19:54:57.713 INFO:teuthology.orchestra.run.smithi146.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.5 MB/s | 44 kB 00:00 2024-01-21T19:54:57.738 INFO:teuthology.orchestra.run.smithi146.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2024-01-21T19:54:57.763 INFO:teuthology.orchestra.run.smithi146.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.3 MB/s | 213 kB 00:00 2024-01-21T19:54:57.797 INFO:teuthology.orchestra.run.smithi146.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2024-01-21T19:54:57.830 INFO:teuthology.orchestra.run.smithi146.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 11 MB/s | 368 kB 00:00 2024-01-21T19:54:57.856 INFO:teuthology.orchestra.run.smithi146.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2024-01-21T19:54:58.039 INFO:teuthology.orchestra.run.smithi146.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 18 MB/s | 13 MB 00:00 2024-01-21T19:54:58.194 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-01-21T19:54:58.197 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:54:58.197 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-01-21T19:54:58.197 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:54:58.197 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-01-21T19:54:58.197 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-01-21T19:54:58.198 INFO:teuthology.orchestra.run.smithi053.stdout:Upgrading: 2024-01-21T19:54:58.198 INFO:teuthology.orchestra.run.smithi053.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-21T19:54:58.198 INFO:teuthology.orchestra.run.smithi053.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-01-21T19:54:58.198 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-01-21T19:54:58.198 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-01-21T19:54:58.198 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-01-21T19:54:58.198 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-01-21T19:54:58.198 INFO:teuthology.orchestra.run.smithi053.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-01-21T19:54:58.198 INFO:teuthology.orchestra.run.smithi053.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-01-21T19:54:58.198 INFO:teuthology.orchestra.run.smithi053.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-01-21T19:54:58.198 INFO:teuthology.orchestra.run.smithi053.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-01-21T19:54:58.198 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-01-21T19:54:58.198 INFO:teuthology.orchestra.run.smithi053.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-01-21T19:54:58.198 INFO:teuthology.orchestra.run.smithi053.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-01-21T19:54:58.198 INFO:teuthology.orchestra.run.smithi053.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-01-21T19:54:58.199 INFO:teuthology.orchestra.run.smithi053.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-01-21T19:54:58.199 INFO:teuthology.orchestra.run.smithi053.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-01-21T19:54:58.199 INFO:teuthology.orchestra.run.smithi053.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-01-21T19:54:58.199 INFO:teuthology.orchestra.run.smithi053.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-01-21T19:54:58.199 INFO:teuthology.orchestra.run.smithi053.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-01-21T19:54:58.199 INFO:teuthology.orchestra.run.smithi053.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-01-21T19:54:58.199 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-01-21T19:54:58.199 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-01-21T19:54:58.199 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-01-21T19:54:58.199 INFO:teuthology.orchestra.run.smithi053.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-01-21T19:54:58.199 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-01-21T19:54:58.199 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-01-21T19:54:58.199 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-01-21T19:54:58.199 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:54:58.199 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-01-21T19:54:58.200 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:54:58.200 INFO:teuthology.orchestra.run.smithi053.stdout:Install 25 Packages 2024-01-21T19:54:58.200 INFO:teuthology.orchestra.run.smithi053.stdout:Upgrade 2 Packages 2024-01-21T19:54:58.200 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:54:58.200 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 59 M 2024-01-21T19:54:58.200 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-01-21T19:54:58.323 INFO:teuthology.orchestra.run.smithi146.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 24 MB/s | 24 MB 00:00 2024-01-21T19:54:58.348 INFO:teuthology.orchestra.run.smithi146.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 295 kB/s | 145 kB 00:00 2024-01-21T19:54:58.407 INFO:teuthology.orchestra.run.smithi146.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 755 kB/s | 44 kB 00:00 2024-01-21T19:54:58.432 INFO:teuthology.orchestra.run.smithi146.stdout:(16/27): librdkafka-1.6.1-1.el8.x86_64.rpm 1.7 MB/s | 677 kB 00:00 2024-01-21T19:54:58.549 INFO:teuthology.orchestra.run.smithi146.stdout:(17/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.1 MB/s | 259 kB 00:00 2024-01-21T19:54:58.574 INFO:teuthology.orchestra.run.smithi146.stdout:(18/27): daxctl-libs-71.1-7.el8.x86_64.rpm 252 kB/s | 42 kB 00:00 2024-01-21T19:54:58.600 INFO:teuthology.orchestra.run.smithi146.stdout:(19/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 281 kB/s | 47 kB 00:00 2024-01-21T19:54:58.614 INFO:teuthology.orchestra.run.smithi053.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 40 MB/s | 6.0 MB 00:00 2024-01-21T19:54:58.631 INFO:teuthology.orchestra.run.smithi053.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 1.5 MB/s | 25 kB 00:00 2024-01-21T19:54:58.642 INFO:teuthology.orchestra.run.smithi146.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 868 kB/s | 80 kB 00:00 2024-01-21T19:54:58.664 INFO:teuthology.orchestra.run.smithi053.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 23 MB/s | 801 kB 00:00 2024-01-21T19:54:58.699 INFO:teuthology.orchestra.run.smithi053.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2024-01-21T19:54:58.750 INFO:teuthology.orchestra.run.smithi146.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 663 kB/s | 116 kB 00:00 2024-01-21T19:54:58.776 INFO:teuthology.orchestra.run.smithi146.stdout:(22/27): leveldb-1.22-1.el8.x86_64.rpm 1.3 MB/s | 181 kB 00:00 2024-01-21T19:54:58.801 INFO:teuthology.orchestra.run.smithi146.stdout:(23/27): liboath-2.6.2-3.el8.x86_64.rpm 1.1 MB/s | 59 kB 00:00 2024-01-21T19:54:58.826 INFO:teuthology.orchestra.run.smithi146.stdout:(24/27): gperftools-libs-2.7-9.el8.x86_64.rpm 1.3 MB/s | 306 kB 00:00 2024-01-21T19:54:58.832 INFO:teuthology.orchestra.run.smithi053.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 31 MB/s | 4.1 MB 00:00 2024-01-21T19:54:58.851 INFO:teuthology.orchestra.run.smithi146.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 995 kB/s | 75 kB 00:00 2024-01-21T19:54:58.857 INFO:teuthology.orchestra.run.smithi053.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.7 MB/s | 44 kB 00:00 2024-01-21T19:54:58.891 INFO:teuthology.orchestra.run.smithi053.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 2.3 MB/s | 80 kB 00:00 2024-01-21T19:54:58.916 INFO:teuthology.orchestra.run.smithi053.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.4 MB/s | 213 kB 00:00 2024-01-21T19:54:58.927 INFO:teuthology.orchestra.run.smithi146.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 30 MB/s | 3.7 MB 00:00 2024-01-21T19:54:58.950 INFO:teuthology.orchestra.run.smithi053.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2024-01-21T19:54:59.044 INFO:teuthology.orchestra.run.smithi146.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 19 MB/s | 4.0 MB 00:00 2024-01-21T19:54:59.049 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:54:59.050 INFO:teuthology.orchestra.run.smithi146.stdout:Total 29 MB/s | 59 MB 00:02 2024-01-21T19:54:59.109 INFO:teuthology.orchestra.run.smithi053.stdout:(10/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 20 MB/s | 13 MB 00:00 2024-01-21T19:54:59.113 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-01-21T19:54:59.134 INFO:teuthology.orchestra.run.smithi053.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2024-01-21T19:54:59.159 INFO:teuthology.orchestra.run.smithi053.stdout:(12/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 1.7 MB/s | 368 kB 00:00 2024-01-21T19:54:59.160 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-01-21T19:54:59.160 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-01-21T19:54:59.451 INFO:teuthology.orchestra.run.smithi053.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 24 MB/s | 24 MB 00:00 2024-01-21T19:54:59.476 INFO:teuthology.orchestra.run.smithi053.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 424 kB/s | 145 kB 00:00 2024-01-21T19:54:59.535 INFO:teuthology.orchestra.run.smithi053.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 750 kB/s | 44 kB 00:00 2024-01-21T19:54:59.569 INFO:teuthology.orchestra.run.smithi053.stdout:(16/27): librdkafka-1.6.1-1.el8.x86_64.rpm 1.6 MB/s | 677 kB 00:00 2024-01-21T19:54:59.571 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-01-21T19:54:59.574 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-01-21T19:54:59.636 INFO:teuthology.orchestra.run.smithi053.stdout:(17/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.4 MB/s | 259 kB 00:00 2024-01-21T19:54:59.753 INFO:teuthology.orchestra.run.smithi053.stdout:(18/27): daxctl-libs-71.1-7.el8.x86_64.rpm 194 kB/s | 42 kB 00:00 2024-01-21T19:54:59.778 INFO:teuthology.orchestra.run.smithi053.stdout:(19/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 225 kB/s | 47 kB 00:00 2024-01-21T19:54:59.820 INFO:teuthology.orchestra.run.smithi053.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 435 kB/s | 80 kB 00:00 2024-01-21T19:55:00.876 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-01-21T19:55:01.104 INFO:teuthology.orchestra.run.smithi053.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 86 kB/s | 116 kB 00:01 2024-01-21T19:55:01.120 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-01-21T19:55:01.221 INFO:teuthology.orchestra.run.smithi053.stdout:(22/27): leveldb-1.22-1.el8.x86_64.rpm 129 kB/s | 181 kB 00:01 2024-01-21T19:55:01.316 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-01-21T19:55:01.380 INFO:teuthology.orchestra.run.smithi053.stdout:(23/27): gperftools-libs-2.7-9.el8.x86_64.rpm 191 kB/s | 306 kB 00:01 2024-01-21T19:55:01.429 INFO:teuthology.orchestra.run.smithi053.stdout:(24/27): liboath-2.6.2-3.el8.x86_64.rpm 181 kB/s | 59 kB 00:00 2024-01-21T19:55:01.494 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-21T19:55:01.522 INFO:teuthology.orchestra.run.smithi053.stdout:(25/27): librados2-16.2.4-0.el8.x86_64.rpm 26 MB/s | 3.7 MB 00:00 2024-01-21T19:55:01.547 INFO:teuthology.orchestra.run.smithi053.stdout:(26/27): libunwind-1.3.1-3.el8.x86_64.rpm 230 kB/s | 75 kB 00:00 2024-01-21T19:55:01.614 INFO:teuthology.orchestra.run.smithi053.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 22 MB/s | 4.0 MB 00:00 2024-01-21T19:55:01.619 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:55:01.619 INFO:teuthology.orchestra.run.smithi053.stdout:Total 17 MB/s | 59 MB 00:03 2024-01-21T19:55:01.688 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-01-21T19:55:01.741 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-01-21T19:55:01.741 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-01-21T19:55:01.904 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-21T19:55:02.069 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-21T19:55:02.155 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-21T19:55:02.194 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-01-21T19:55:02.198 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-01-21T19:55:02.379 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-01-21T19:55:02.569 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-21T19:55:02.631 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-21T19:55:02.871 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-01-21T19:55:03.387 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-01-21T19:55:03.436 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-01-21T19:55:03.594 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-21T19:55:03.633 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-01-21T19:55:03.670 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-21T19:55:03.857 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-01-21T19:55:03.864 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-01-21T19:55:04.044 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-21T19:55:04.051 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-21T19:55:04.094 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-21T19:55:04.282 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-01-21T19:55:04.478 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-21T19:55:04.494 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-01-21T19:55:04.660 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-21T19:55:04.758 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-21T19:55:05.030 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-01-21T19:55:05.201 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-21T19:55:05.203 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-21T19:55:05.269 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-21T19:55:05.340 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-21T19:55:05.506 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-01-21T19:55:05.516 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-21T19:55:05.935 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-21T19:55:06.077 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-21T19:55:06.088 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-01-21T19:55:06.177 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-21T19:55:06.368 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-21T19:55:06.372 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-01-21T19:55:06.435 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-21T19:55:06.612 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-01-21T19:55:06.665 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-01-21T19:55:06.768 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-21T19:55:06.826 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-21T19:55:06.831 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-21T19:55:06.909 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-21T19:55:07.069 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-01-21T19:55:07.075 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-01-21T19:55:07.276 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-01-21T19:55:07.277 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-01-21T19:55:07.418 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-21T19:55:07.459 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-01-21T19:55:07.512 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-21T19:55:07.646 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-21T19:55:07.677 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-01-21T19:55:08.110 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-21T19:55:08.311 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-21T19:55:08.413 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-21T19:55:08.613 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-01-21T19:55:08.814 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-01-21T19:55:08.967 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-21T19:55:09.052 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-21T19:55:09.279 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-01-21T19:55:09.477 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-01-21T19:55:09.657 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-01-21T19:55:09.893 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-01-21T19:55:10.248 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-21T19:55:10.543 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-21T19:55:11.293 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-21T19:55:11.578 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-21T19:55:11.716 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-21T19:55:11.867 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-21T19:55:12.649 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-21T19:55:12.901 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-21T19:55:13.639 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-21T19:55:14.009 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-21T19:55:14.122 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-21T19:55:14.291 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-21T19:55:31.176 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-21T19:55:31.177 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /sys 2024-01-21T19:55:31.177 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /proc 2024-01-21T19:55:31.177 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /dev 2024-01-21T19:55:31.177 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /run 2024-01-21T19:55:31.177 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /mnt 2024-01-21T19:55:31.177 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /var/tmp 2024-01-21T19:55:31.177 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /home 2024-01-21T19:55:31.177 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /tmp 2024-01-21T19:55:31.177 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /dev 2024-01-21T19:55:31.177 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:55:32.516 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-21T19:55:32.586 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-21T19:55:32.587 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-21T19:55:32.817 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-21T19:55:32.817 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-21T19:55:33.512 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-21T19:55:33.512 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-01-21T19:55:33.512 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-01-21T19:55:33.512 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-01-21T19:55:33.512 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-01-21T19:55:33.512 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-01-21T19:55:33.512 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-01-21T19:55:33.513 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-01-21T19:55:33.513 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-01-21T19:55:33.513 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-01-21T19:55:33.513 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-01-21T19:55:33.513 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-01-21T19:55:33.513 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-01-21T19:55:33.513 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-01-21T19:55:33.513 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-01-21T19:55:33.513 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-01-21T19:55:33.513 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-01-21T19:55:33.513 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-01-21T19:55:33.513 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-01-21T19:55:33.513 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-01-21T19:55:33.513 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-01-21T19:55:33.513 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-01-21T19:55:33.513 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-01-21T19:55:33.514 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-01-21T19:55:33.514 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-01-21T19:55:33.514 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-01-21T19:55:33.514 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-01-21T19:55:33.514 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-01-21T19:55:33.514 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-01-21T19:55:34.250 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-21T19:55:34.250 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /sys 2024-01-21T19:55:34.250 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /proc 2024-01-21T19:55:34.250 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /dev 2024-01-21T19:55:34.250 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /run 2024-01-21T19:55:34.250 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /mnt 2024-01-21T19:55:34.250 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /var/tmp 2024-01-21T19:55:34.250 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /home 2024-01-21T19:55:34.250 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /tmp 2024-01-21T19:55:34.250 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /dev 2024-01-21T19:55:34.250 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:55:34.564 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-01-21T19:55:34.564 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:55:34.564 INFO:teuthology.orchestra.run.smithi146.stdout:Upgraded: 2024-01-21T19:55:34.564 INFO:teuthology.orchestra.run.smithi146.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-01-21T19:55:34.564 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-01-21T19:55:34.564 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-01-21T19:55:34.564 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-01-21T19:55:34.564 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-01-21T19:55:34.564 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-01-21T19:55:34.564 INFO:teuthology.orchestra.run.smithi146.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-01-21T19:55:34.564 INFO:teuthology.orchestra.run.smithi146.stdout: fmt-6.2.1-1.el8.x86_64 2024-01-21T19:55:34.565 INFO:teuthology.orchestra.run.smithi146.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-01-21T19:55:34.565 INFO:teuthology.orchestra.run.smithi146.stdout: leveldb-1.22-1.el8.x86_64 2024-01-21T19:55:34.565 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-01-21T19:55:34.565 INFO:teuthology.orchestra.run.smithi146.stdout: liboath-2.6.2-3.el8.x86_64 2024-01-21T19:55:34.565 INFO:teuthology.orchestra.run.smithi146.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-01-21T19:55:34.565 INFO:teuthology.orchestra.run.smithi146.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-01-21T19:55:34.565 INFO:teuthology.orchestra.run.smithi146.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-01-21T19:55:34.565 INFO:teuthology.orchestra.run.smithi146.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-01-21T19:55:34.565 INFO:teuthology.orchestra.run.smithi146.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-01-21T19:55:34.565 INFO:teuthology.orchestra.run.smithi146.stdout: libunwind-1.3.1-3.el8.x86_64 2024-01-21T19:55:34.565 INFO:teuthology.orchestra.run.smithi146.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-01-21T19:55:34.565 INFO:teuthology.orchestra.run.smithi146.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-01-21T19:55:34.565 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-01-21T19:55:34.565 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-01-21T19:55:34.565 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-01-21T19:55:34.565 INFO:teuthology.orchestra.run.smithi146.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-01-21T19:55:34.566 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-01-21T19:55:34.566 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-01-21T19:55:34.566 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-01-21T19:55:34.566 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:55:34.566 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-01-21T19:55:34.884 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-test 2024-01-21T19:55:35.609 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:00:42 ago on Sun 21 Jan 2024 07:54:53 PM UTC. 2024-01-21T19:55:35.631 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-21T19:55:35.697 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-21T19:55:35.697 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-21T19:55:35.908 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-21T19:55:35.908 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-21T19:55:36.618 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-21T19:55:36.618 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-01-21T19:55:36.618 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-01-21T19:55:36.618 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-01-21T19:55:36.618 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-01-21T19:55:36.618 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-01-21T19:55:36.618 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-01-21T19:55:36.618 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-01-21T19:55:36.619 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-01-21T19:55:36.619 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-01-21T19:55:36.619 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-01-21T19:55:36.619 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-01-21T19:55:36.619 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-01-21T19:55:36.619 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-01-21T19:55:36.619 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-01-21T19:55:36.619 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-01-21T19:55:36.619 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-01-21T19:55:36.619 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-01-21T19:55:36.619 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-01-21T19:55:36.619 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-01-21T19:55:36.619 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-01-21T19:55:36.619 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-01-21T19:55:36.620 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-01-21T19:55:36.620 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-01-21T19:55:36.620 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-01-21T19:55:36.620 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-01-21T19:55:36.620 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-01-21T19:55:36.620 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-01-21T19:55:36.620 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-01-21T19:55:37.093 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-01-21T19:55:37.095 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:55:37.095 INFO:teuthology.orchestra.run.smithi146.stdout: Package Architecture Version Repository Size 2024-01-21T19:55:37.095 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:55:37.095 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-01-21T19:55:37.095 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-01-21T19:55:37.095 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-01-21T19:55:37.095 INFO:teuthology.orchestra.run.smithi146.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-01-21T19:55:37.095 INFO:teuthology.orchestra.run.smithi146.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-01-21T19:55:37.095 INFO:teuthology.orchestra.run.smithi146.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-01-21T19:55:37.095 INFO:teuthology.orchestra.run.smithi146.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-01-21T19:55:37.095 INFO:teuthology.orchestra.run.smithi146.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-01-21T19:55:37.095 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:55:37.096 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-01-21T19:55:37.096 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:55:37.096 INFO:teuthology.orchestra.run.smithi146.stdout:Install 6 Packages 2024-01-21T19:55:37.096 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:55:37.096 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 53 M 2024-01-21T19:55:37.096 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 229 M 2024-01-21T19:55:37.096 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-01-21T19:55:37.328 INFO:teuthology.orchestra.run.smithi146.stdout:(1/6): jq-1.6-8.el8.x86_64.rpm 1.0 MB/s | 203 kB 00:00 2024-01-21T19:55:37.353 INFO:teuthology.orchestra.run.smithi146.stdout:(2/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 787 kB/s | 169 kB 00:00 2024-01-21T19:55:37.378 INFO:teuthology.orchestra.run.smithi146.stdout:(3/6): oniguruma-6.8.2-3.el8.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2024-01-21T19:55:37.428 INFO:teuthology.orchestra.run.smithi146.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 71 kB 00:00 2024-01-21T19:55:37.552 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-01-21T19:55:37.552 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:55:37.552 INFO:teuthology.orchestra.run.smithi053.stdout:Upgraded: 2024-01-21T19:55:37.552 INFO:teuthology.orchestra.run.smithi053.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-01-21T19:55:37.552 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-01-21T19:55:37.552 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-01-21T19:55:37.552 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-01-21T19:55:37.553 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-01-21T19:55:37.553 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-01-21T19:55:37.553 INFO:teuthology.orchestra.run.smithi053.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-01-21T19:55:37.553 INFO:teuthology.orchestra.run.smithi053.stdout: fmt-6.2.1-1.el8.x86_64 2024-01-21T19:55:37.553 INFO:teuthology.orchestra.run.smithi053.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-01-21T19:55:37.553 INFO:teuthology.orchestra.run.smithi053.stdout: leveldb-1.22-1.el8.x86_64 2024-01-21T19:55:37.553 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-01-21T19:55:37.553 INFO:teuthology.orchestra.run.smithi053.stdout: liboath-2.6.2-3.el8.x86_64 2024-01-21T19:55:37.553 INFO:teuthology.orchestra.run.smithi053.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-01-21T19:55:37.553 INFO:teuthology.orchestra.run.smithi053.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-01-21T19:55:37.553 INFO:teuthology.orchestra.run.smithi053.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-01-21T19:55:37.553 INFO:teuthology.orchestra.run.smithi053.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-01-21T19:55:37.553 INFO:teuthology.orchestra.run.smithi053.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-01-21T19:55:37.553 INFO:teuthology.orchestra.run.smithi053.stdout: libunwind-1.3.1-3.el8.x86_64 2024-01-21T19:55:37.553 INFO:teuthology.orchestra.run.smithi053.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-01-21T19:55:37.554 INFO:teuthology.orchestra.run.smithi053.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-01-21T19:55:37.554 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-01-21T19:55:37.554 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-01-21T19:55:37.554 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-01-21T19:55:37.554 INFO:teuthology.orchestra.run.smithi053.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-01-21T19:55:37.554 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-01-21T19:55:37.554 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-01-21T19:55:37.554 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-01-21T19:55:37.554 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:55:37.554 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-01-21T19:55:37.865 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-test 2024-01-21T19:55:37.945 INFO:teuthology.orchestra.run.smithi146.stdout:(5/6): ceph-test-16.2.4-0.el8.x86_64.rpm 64 MB/s | 52 MB 00:00 2024-01-21T19:55:38.037 INFO:teuthology.orchestra.run.smithi146.stdout:(6/6): socat-1.7.4.1-1.el8.x86_64.rpm 472 kB/s | 323 kB 00:00 2024-01-21T19:55:38.039 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:55:38.039 INFO:teuthology.orchestra.run.smithi146.stdout:Total 56 MB/s | 53 MB 00:00 2024-01-21T19:55:38.081 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-01-21T19:55:38.103 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-01-21T19:55:38.103 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-01-21T19:55:38.422 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-01-21T19:55:38.425 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-01-21T19:55:38.624 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:00:44 ago on Sun 21 Jan 2024 07:54:54 PM UTC. 2024-01-21T19:55:38.931 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-01-21T19:55:39.141 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-01-21T19:55:39.370 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-01-21T19:55:39.555 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-01-21T19:55:39.634 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-01-21T19:55:39.837 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-01-21T19:55:40.013 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-21T19:55:40.228 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-01-21T19:55:40.229 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:55:40.229 INFO:teuthology.orchestra.run.smithi053.stdout: Package Architecture Version Repository Size 2024-01-21T19:55:40.229 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:55:40.229 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-01-21T19:55:40.230 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-01-21T19:55:40.230 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-01-21T19:55:40.230 INFO:teuthology.orchestra.run.smithi053.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-01-21T19:55:40.230 INFO:teuthology.orchestra.run.smithi053.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-01-21T19:55:40.230 INFO:teuthology.orchestra.run.smithi053.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-01-21T19:55:40.230 INFO:teuthology.orchestra.run.smithi053.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-01-21T19:55:40.230 INFO:teuthology.orchestra.run.smithi053.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-01-21T19:55:40.230 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:55:40.230 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-01-21T19:55:40.230 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:55:40.230 INFO:teuthology.orchestra.run.smithi053.stdout:Install 6 Packages 2024-01-21T19:55:40.230 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:55:40.231 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 53 M 2024-01-21T19:55:40.231 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 229 M 2024-01-21T19:55:40.231 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-01-21T19:55:40.337 INFO:teuthology.orchestra.run.smithi053.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 2.6 MB/s | 169 kB 00:00 2024-01-21T19:55:40.495 INFO:teuthology.orchestra.run.smithi053.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 917 kB/s | 203 kB 00:00 2024-01-21T19:55:40.554 INFO:teuthology.orchestra.run.smithi053.stdout:(3/6): oniguruma-6.8.2-3.el8.x86_64.rpm 868 kB/s | 188 kB 00:00 2024-01-21T19:55:40.587 INFO:teuthology.orchestra.run.smithi053.stdout:(4/6): socat-1.7.4.1-1.el8.x86_64.rpm 3.4 MB/s | 323 kB 00:00 2024-01-21T19:55:40.604 INFO:teuthology.orchestra.run.smithi053.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 71 kB 00:00 2024-01-21T19:55:41.191 INFO:teuthology.orchestra.run.smithi053.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 57 MB/s | 52 MB 00:00 2024-01-21T19:55:41.192 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:55:41.192 INFO:teuthology.orchestra.run.smithi053.stdout:Total 55 MB/s | 53 MB 00:00 2024-01-21T19:55:41.234 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-01-21T19:55:41.257 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-01-21T19:55:41.257 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-01-21T19:55:41.582 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-01-21T19:55:41.585 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-01-21T19:55:42.121 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-01-21T19:55:42.368 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-01-21T19:55:42.552 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-01-21T19:55:42.713 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-01-21T19:55:42.793 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-01-21T19:55:42.988 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-01-21T19:55:43.154 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-21T19:55:44.840 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-21T19:55:45.086 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-21T19:55:45.399 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-21T19:55:45.399 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-01-21T19:55:45.399 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-01-21T19:55:45.399 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-01-21T19:55:45.399 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-01-21T19:55:45.399 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-01-21T19:55:45.833 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-01-21T19:55:45.833 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:55:45.833 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-01-21T19:55:45.833 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-01-21T19:55:45.833 INFO:teuthology.orchestra.run.smithi146.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-01-21T19:55:45.833 INFO:teuthology.orchestra.run.smithi146.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-01-21T19:55:45.833 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:55:45.833 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-01-21T19:55:46.085 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph 2024-01-21T19:55:46.816 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:00:53 ago on Sun 21 Jan 2024 07:54:53 PM UTC. 2024-01-21T19:55:48.289 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-01-21T19:55:48.292 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:55:48.292 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-01-21T19:55:48.293 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:55:48.293 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-01-21T19:55:48.293 INFO:teuthology.orchestra.run.smithi146.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-01-21T19:55:48.293 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-01-21T19:55:48.293 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-01-21T19:55:48.293 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-01-21T19:55:48.293 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-01-21T19:55:48.293 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-01-21T19:55:48.293 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-01-21T19:55:48.293 INFO:teuthology.orchestra.run.smithi146.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-01-21T19:55:48.293 INFO:teuthology.orchestra.run.smithi146.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-01-21T19:55:48.293 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-01-21T19:55:48.293 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-01-21T19:55:48.293 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-01-21T19:55:48.294 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-01-21T19:55:48.294 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-01-21T19:55:48.294 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-01-21T19:55:48.294 INFO:teuthology.orchestra.run.smithi146.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-01-21T19:55:48.294 INFO:teuthology.orchestra.run.smithi146.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-01-21T19:55:48.294 INFO:teuthology.orchestra.run.smithi146.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-01-21T19:55:48.294 INFO:teuthology.orchestra.run.smithi146.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-01-21T19:55:48.294 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-01-21T19:55:48.294 INFO:teuthology.orchestra.run.smithi146.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-01-21T19:55:48.294 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-01-21T19:55:48.294 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-01-21T19:55:48.294 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-01-21T19:55:48.294 INFO:teuthology.orchestra.run.smithi146.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-01-21T19:55:48.294 INFO:teuthology.orchestra.run.smithi146.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-01-21T19:55:48.294 INFO:teuthology.orchestra.run.smithi146.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-01-21T19:55:48.295 INFO:teuthology.orchestra.run.smithi146.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-01-21T19:55:48.295 INFO:teuthology.orchestra.run.smithi146.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-01-21T19:55:48.295 INFO:teuthology.orchestra.run.smithi146.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-01-21T19:55:48.295 INFO:teuthology.orchestra.run.smithi146.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-01-21T19:55:48.295 INFO:teuthology.orchestra.run.smithi146.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-01-21T19:55:48.295 INFO:teuthology.orchestra.run.smithi146.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-01-21T19:55:48.295 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:55:48.295 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-01-21T19:55:48.295 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:55:48.295 INFO:teuthology.orchestra.run.smithi146.stdout:Install 32 Packages 2024-01-21T19:55:48.295 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:55:48.296 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 31 M 2024-01-21T19:55:48.296 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 111 M 2024-01-21T19:55:48.296 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-01-21T19:55:48.347 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-21T19:55:48.603 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-21T19:55:48.664 INFO:teuthology.orchestra.run.smithi146.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 98 kB/s | 6.3 kB 00:00 2024-01-21T19:55:48.725 INFO:teuthology.orchestra.run.smithi146.stdout:(2/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 19 MB/s | 2.4 MB 00:00 2024-01-21T19:55:48.773 INFO:teuthology.orchestra.run.smithi146.stdout:(3/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 9.0 MB/s | 1.6 MB 00:00 2024-01-21T19:55:48.798 INFO:teuthology.orchestra.run.smithi146.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.7 MB/s | 220 kB 00:00 2024-01-21T19:55:48.882 INFO:teuthology.orchestra.run.smithi146.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 23 MB/s | 5.0 MB 00:00 2024-01-21T19:55:48.902 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-21T19:55:48.902 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-01-21T19:55:48.902 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-01-21T19:55:48.902 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-01-21T19:55:48.902 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-01-21T19:55:48.903 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-01-21T19:55:49.132 INFO:teuthology.orchestra.run.smithi146.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 45 MB/s | 18 MB 00:00 2024-01-21T19:55:49.149 INFO:teuthology.orchestra.run.smithi146.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 145 kB/s | 39 kB 00:00 2024-01-21T19:55:49.174 INFO:teuthology.orchestra.run.smithi146.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 417 kB/s | 157 kB 00:00 2024-01-21T19:55:49.258 INFO:teuthology.orchestra.run.smithi146.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 525 kB/s | 57 kB 00:00 2024-01-21T19:55:49.300 INFO:teuthology.orchestra.run.smithi146.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 3.6 MB/s | 457 kB 00:00 2024-01-21T19:55:49.325 INFO:teuthology.orchestra.run.smithi146.stdout:(11/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 537 kB/s | 103 kB 00:00 2024-01-21T19:55:49.329 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-01-21T19:55:49.330 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:55:49.330 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-01-21T19:55:49.330 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-01-21T19:55:49.330 INFO:teuthology.orchestra.run.smithi053.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-01-21T19:55:49.330 INFO:teuthology.orchestra.run.smithi053.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-01-21T19:55:49.330 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:55:49.331 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-01-21T19:55:49.525 INFO:teuthology.orchestra.run.smithi146.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 931 kB/s | 249 kB 00:00 2024-01-21T19:55:49.542 INFO:teuthology.orchestra.run.smithi146.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 503 kB/s | 109 kB 00:00 2024-01-21T19:55:49.567 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph 2024-01-21T19:55:49.634 INFO:teuthology.orchestra.run.smithi146.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 401 kB/s | 44 kB 00:00 2024-01-21T19:55:49.670 INFO:teuthology.orchestra.run.smithi146.stdout:(15/32): python3-cryptography-3.2.1-7.el8.x86_6 1.7 MB/s | 641 kB 00:00 2024-01-21T19:55:49.695 INFO:teuthology.orchestra.run.smithi146.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.2 MB/s | 185 kB 00:00 2024-01-21T19:55:49.729 INFO:teuthology.orchestra.run.smithi146.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.8 MB/s | 173 kB 00:00 2024-01-21T19:55:49.779 INFO:teuthology.orchestra.run.smithi146.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 133 kB/s | 11 kB 00:00 2024-01-21T19:55:49.804 INFO:teuthology.orchestra.run.smithi146.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 233 kB/s | 18 kB 00:00 2024-01-21T19:55:49.830 INFO:teuthology.orchestra.run.smithi146.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 972 kB/s | 49 kB 00:00 2024-01-21T19:55:49.855 INFO:teuthology.orchestra.run.smithi146.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 1.1 MB/s | 59 kB 00:00 2024-01-21T19:55:49.888 INFO:teuthology.orchestra.run.smithi146.stdout:(22/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.7 MB/s | 384 kB 00:00 2024-01-21T19:55:49.914 INFO:teuthology.orchestra.run.smithi146.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 266 kB/s | 16 kB 00:00 2024-01-21T19:55:49.939 INFO:teuthology.orchestra.run.smithi146.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 373 kB/s | 19 kB 00:00 2024-01-21T19:55:49.964 INFO:teuthology.orchestra.run.smithi146.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 479 kB/s | 24 kB 00:00 2024-01-21T19:55:49.990 INFO:teuthology.orchestra.run.smithi146.stdout:(26/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.7 MB/s | 283 kB 00:00 2024-01-21T19:55:50.040 INFO:teuthology.orchestra.run.smithi146.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 290 kB/s | 29 kB 00:00 2024-01-21T19:55:50.057 INFO:teuthology.orchestra.run.smithi146.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 298 kB/s | 27 kB 00:00 2024-01-21T19:55:50.082 INFO:teuthology.orchestra.run.smithi146.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.7 MB/s | 255 kB 00:00 2024-01-21T19:55:50.107 INFO:teuthology.orchestra.run.smithi146.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.7 MB/s | 85 kB 00:00 2024-01-21T19:55:50.124 INFO:teuthology.orchestra.run.smithi146.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 550 kB/s | 23 kB 00:00 2024-01-21T19:55:50.149 INFO:teuthology.orchestra.run.smithi146.stdout:(32/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.3 MB/s | 251 kB 00:00 2024-01-21T19:55:50.150 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:55:50.151 INFO:teuthology.orchestra.run.smithi146.stdout:Total 17 MB/s | 31 MB 00:01 2024-01-21T19:55:50.234 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-01-21T19:55:50.272 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-01-21T19:55:50.272 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-01-21T19:55:50.324 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:00:56 ago on Sun 21 Jan 2024 07:54:54 PM UTC. 2024-01-21T19:55:50.607 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-01-21T19:55:50.610 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-01-21T19:55:51.794 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-01-21T19:55:51.947 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-01-21T19:55:51.950 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:55:51.950 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-01-21T19:55:51.950 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:55:51.950 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-01-21T19:55:51.951 INFO:teuthology.orchestra.run.smithi053.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-01-21T19:55:51.951 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-01-21T19:55:51.951 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-01-21T19:55:51.951 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-01-21T19:55:51.951 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-01-21T19:55:51.951 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-01-21T19:55:51.951 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-01-21T19:55:51.951 INFO:teuthology.orchestra.run.smithi053.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-01-21T19:55:51.951 INFO:teuthology.orchestra.run.smithi053.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-01-21T19:55:51.951 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-01-21T19:55:51.951 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-01-21T19:55:51.951 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-01-21T19:55:51.951 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-01-21T19:55:51.951 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-01-21T19:55:51.951 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-01-21T19:55:51.952 INFO:teuthology.orchestra.run.smithi053.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-01-21T19:55:51.952 INFO:teuthology.orchestra.run.smithi053.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-01-21T19:55:51.952 INFO:teuthology.orchestra.run.smithi053.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-01-21T19:55:51.952 INFO:teuthology.orchestra.run.smithi053.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-01-21T19:55:51.952 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-01-21T19:55:51.952 INFO:teuthology.orchestra.run.smithi053.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-01-21T19:55:51.952 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-01-21T19:55:51.952 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-01-21T19:55:51.952 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-01-21T19:55:51.952 INFO:teuthology.orchestra.run.smithi053.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-01-21T19:55:51.952 INFO:teuthology.orchestra.run.smithi053.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-01-21T19:55:51.952 INFO:teuthology.orchestra.run.smithi053.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-01-21T19:55:51.953 INFO:teuthology.orchestra.run.smithi053.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-01-21T19:55:51.953 INFO:teuthology.orchestra.run.smithi053.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-01-21T19:55:51.953 INFO:teuthology.orchestra.run.smithi053.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-01-21T19:55:51.953 INFO:teuthology.orchestra.run.smithi053.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-01-21T19:55:51.953 INFO:teuthology.orchestra.run.smithi053.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-01-21T19:55:51.953 INFO:teuthology.orchestra.run.smithi053.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-01-21T19:55:51.953 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:55:51.953 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-01-21T19:55:51.953 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:55:51.953 INFO:teuthology.orchestra.run.smithi053.stdout:Install 32 Packages 2024-01-21T19:55:51.953 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:55:51.954 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 31 M 2024-01-21T19:55:51.954 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 111 M 2024-01-21T19:55:51.954 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-01-21T19:55:52.016 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-01-21T19:55:52.227 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-01-21T19:55:52.406 INFO:teuthology.orchestra.run.smithi053.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 156 kB/s | 6.3 kB 00:00 2024-01-21T19:55:52.473 INFO:teuthology.orchestra.run.smithi053.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-01-21T19:55:52.477 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-01-21T19:55:52.524 INFO:teuthology.orchestra.run.smithi053.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 15 MB/s | 2.4 MB 00:00 2024-01-21T19:55:52.548 INFO:teuthology.orchestra.run.smithi053.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.6 MB/s | 220 kB 00:00 2024-01-21T19:55:52.641 INFO:teuthology.orchestra.run.smithi053.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 21 MB/s | 5.0 MB 00:00 2024-01-21T19:55:52.701 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-01-21T19:55:52.865 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-01-21T19:55:52.891 INFO:teuthology.orchestra.run.smithi053.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 44 MB/s | 18 MB 00:00 2024-01-21T19:55:52.916 INFO:teuthology.orchestra.run.smithi053.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 141 kB/s | 39 kB 00:00 2024-01-21T19:55:52.941 INFO:teuthology.orchestra.run.smithi053.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 400 kB/s | 157 kB 00:00 2024-01-21T19:55:53.033 INFO:teuthology.orchestra.run.smithi053.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 487 kB/s | 57 kB 00:00 2024-01-21T19:55:53.075 INFO:teuthology.orchestra.run.smithi053.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 3.3 MB/s | 457 kB 00:00 2024-01-21T19:55:53.078 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-01-21T19:55:53.100 INFO:teuthology.orchestra.run.smithi053.stdout:(11/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 494 kB/s | 103 kB 00:00 2024-01-21T19:55:53.264 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-01-21T19:55:53.318 INFO:teuthology.orchestra.run.smithi053.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 875 kB/s | 249 kB 00:00 2024-01-21T19:55:53.343 INFO:teuthology.orchestra.run.smithi053.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 451 kB/s | 109 kB 00:00 2024-01-21T19:55:53.435 INFO:teuthology.orchestra.run.smithi053.stdout:(14/32): python3-cryptography-3.2.1-7.el8.x86_6 1.7 MB/s | 641 kB 00:00 2024-01-21T19:55:53.483 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-01-21T19:55:53.572 INFO:teuthology.orchestra.run.smithi053.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 172 kB/s | 44 kB 00:00 2024-01-21T19:55:53.602 INFO:teuthology.orchestra.run.smithi053.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 716 kB/s | 185 kB 00:00 2024-01-21T19:55:53.669 INFO:teuthology.orchestra.run.smithi053.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 166 kB/s | 11 kB 00:00 2024-01-21T19:55:53.676 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-01-21T19:55:53.694 INFO:teuthology.orchestra.run.smithi053.stdout:(18/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 666 kB/s | 173 kB 00:00 2024-01-21T19:55:53.736 INFO:teuthology.orchestra.run.smithi053.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 263 kB/s | 18 kB 00:00 2024-01-21T19:55:53.762 INFO:teuthology.orchestra.run.smithi053.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 729 kB/s | 49 kB 00:00 2024-01-21T19:55:53.786 INFO:teuthology.orchestra.run.smithi053.stdout:(21/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.8 MB/s | 384 kB 00:00 2024-01-21T19:55:53.871 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-01-21T19:55:53.912 INFO:teuthology.orchestra.run.smithi053.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 334 kB/s | 59 kB 00:00 2024-01-21T19:55:53.937 INFO:teuthology.orchestra.run.smithi053.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 104 kB/s | 16 kB 00:00 2024-01-21T19:55:53.963 INFO:teuthology.orchestra.run.smithi053.stdout:(24/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.4 MB/s | 283 kB 00:00 2024-01-21T19:55:53.988 INFO:teuthology.orchestra.run.smithi053.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 248 kB/s | 19 kB 00:00 2024-01-21T19:55:54.013 INFO:teuthology.orchestra.run.smithi053.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 317 kB/s | 24 kB 00:00 2024-01-21T19:55:54.038 INFO:teuthology.orchestra.run.smithi053.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 387 kB/s | 29 kB 00:00 2024-01-21T19:55:54.064 INFO:teuthology.orchestra.run.smithi053.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 365 kB/s | 27 kB 00:00 2024-01-21T19:55:54.109 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-01-21T19:55:54.138 INFO:teuthology.orchestra.run.smithi053.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.5 MB/s | 251 kB 00:00 2024-01-21T19:55:54.173 INFO:teuthology.orchestra.run.smithi053.stdout:(30/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.6 MB/s | 255 kB 00:00 2024-01-21T19:55:54.198 INFO:teuthology.orchestra.run.smithi053.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 638 kB/s | 85 kB 00:00 2024-01-21T19:55:54.214 INFO:teuthology.orchestra.run.smithi053.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 302 kB/s | 23 kB 00:00 2024-01-21T19:55:54.216 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:55:54.216 INFO:teuthology.orchestra.run.smithi053.stdout:Total 14 MB/s | 31 MB 00:02 2024-01-21T19:55:54.301 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-01-21T19:55:54.342 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-01-21T19:55:54.343 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-01-21T19:55:54.367 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-01-21T19:55:54.668 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-01-21T19:55:54.671 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-01-21T19:55:54.753 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-01-21T19:55:54.997 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-01-21T19:55:55.168 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-01-21T19:55:55.426 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-01-21T19:55:55.619 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-21T19:55:55.870 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-01-21T19:55:55.935 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-01-21T19:55:56.063 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-01-21T19:55:56.166 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-01-21T19:55:56.278 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-01-21T19:55:56.363 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-01-21T19:55:56.567 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-01-21T19:55:56.569 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-01-21T19:55:56.793 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-01-21T19:55:56.836 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-01-21T19:55:56.949 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-01-21T19:55:57.087 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-01-21T19:55:57.158 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-01-21T19:55:57.331 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-01-21T19:55:57.356 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-01-21T19:55:57.534 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-01-21T19:55:57.705 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-01-21T19:55:57.762 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-01-21T19:55:58.015 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-01-21T19:55:58.024 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-01-21T19:55:58.254 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-01-21T19:55:58.425 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-01-21T19:55:58.528 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-01-21T19:55:58.629 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-21T19:55:58.964 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-01-21T19:55:59.259 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-01-21T19:55:59.444 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-01-21T19:55:59.716 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-01-21T19:55:59.938 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-21T19:56:00.161 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-21T19:56:00.265 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-01-21T19:56:00.362 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-21T19:56:00.502 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-01-21T19:56:00.715 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-01-21T19:56:00.797 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-21T19:56:00.903 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-01-21T19:56:01.054 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-21T19:56:01.179 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-01-21T19:56:01.275 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-21T19:56:01.454 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-01-21T19:56:01.468 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-21T19:56:01.486 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-21T19:56:01.651 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-21T19:56:01.683 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-01-21T19:56:02.038 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-01-21T19:56:02.056 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-21T19:56:02.056 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-01-21T19:56:02.056 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-01-21T19:56:02.056 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-01-21T19:56:02.056 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-01-21T19:56:02.057 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-01-21T19:56:02.057 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-01-21T19:56:02.057 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-01-21T19:56:02.057 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-01-21T19:56:02.057 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-01-21T19:56:02.057 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-01-21T19:56:02.057 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-01-21T19:56:02.057 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-01-21T19:56:02.057 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-01-21T19:56:02.057 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-01-21T19:56:02.057 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-01-21T19:56:02.057 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-01-21T19:56:02.057 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-21T19:56:02.057 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-01-21T19:56:02.057 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-01-21T19:56:02.058 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-01-21T19:56:02.058 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-01-21T19:56:02.058 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-01-21T19:56:02.058 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-01-21T19:56:02.058 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-01-21T19:56:02.058 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-01-21T19:56:02.058 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-01-21T19:56:02.058 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-01-21T19:56:02.058 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-01-21T19:56:02.059 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-01-21T19:56:02.059 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-01-21T19:56:02.059 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-01-21T19:56:02.326 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-01-21T19:56:02.724 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-01-21T19:56:02.805 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-01-21T19:56:02.805 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:56:02.805 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-01-21T19:56:02.805 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-01-21T19:56:02.805 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-01-21T19:56:02.806 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-01-21T19:56:02.806 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-01-21T19:56:02.806 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-01-21T19:56:02.806 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-01-21T19:56:02.806 INFO:teuthology.orchestra.run.smithi146.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-01-21T19:56:02.806 INFO:teuthology.orchestra.run.smithi146.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-01-21T19:56:02.806 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-01-21T19:56:02.806 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-01-21T19:56:02.806 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-01-21T19:56:02.806 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-01-21T19:56:02.806 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-6.2-6.el8.noarch 2024-01-21T19:56:02.806 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-01-21T19:56:02.806 INFO:teuthology.orchestra.run.smithi146.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-01-21T19:56:02.807 INFO:teuthology.orchestra.run.smithi146.stdout: python3-mako-1.0.6-14.el8.noarch 2024-01-21T19:56:02.807 INFO:teuthology.orchestra.run.smithi146.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-01-21T19:56:02.807 INFO:teuthology.orchestra.run.smithi146.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-01-21T19:56:02.807 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-01-21T19:56:02.807 INFO:teuthology.orchestra.run.smithi146.stdout: python3-portend-2.6-1.el8.noarch 2024-01-21T19:56:02.807 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-01-21T19:56:02.807 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pycparser-2.14-14.el8.noarch 2024-01-21T19:56:02.807 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pytz-2017.2-11.el8.noarch 2024-01-21T19:56:02.807 INFO:teuthology.orchestra.run.smithi146.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-01-21T19:56:02.807 INFO:teuthology.orchestra.run.smithi146.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-01-21T19:56:02.807 INFO:teuthology.orchestra.run.smithi146.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-01-21T19:56:02.807 INFO:teuthology.orchestra.run.smithi146.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-01-21T19:56:02.807 INFO:teuthology.orchestra.run.smithi146.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-01-21T19:56:02.807 INFO:teuthology.orchestra.run.smithi146.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-01-21T19:56:02.808 INFO:teuthology.orchestra.run.smithi146.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-01-21T19:56:02.808 INFO:teuthology.orchestra.run.smithi146.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-01-21T19:56:02.808 INFO:teuthology.orchestra.run.smithi146.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-01-21T19:56:02.808 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:56:02.808 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-01-21T19:56:02.911 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-21T19:56:03.047 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-base 2024-01-21T19:56:03.783 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:10 ago on Sun 21 Jan 2024 07:54:53 PM UTC. 2024-01-21T19:56:04.539 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-21T19:56:04.744 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-21T19:56:05.204 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-21T19:56:05.230 INFO:teuthology.orchestra.run.smithi146.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:56:05.276 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-01-21T19:56:05.277 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-01-21T19:56:05.277 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-01-21T19:56:05.387 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install cephadm 2024-01-21T19:56:05.418 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-21T19:56:05.654 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-21T19:56:05.860 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-21T19:56:05.879 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-21T19:56:06.002 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-21T19:56:06.109 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:13 ago on Sun 21 Jan 2024 07:54:53 PM UTC. 2024-01-21T19:56:06.486 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-21T19:56:06.486 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-01-21T19:56:06.486 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-01-21T19:56:06.486 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-01-21T19:56:06.486 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-01-21T19:56:06.486 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-01-21T19:56:06.487 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-01-21T19:56:06.487 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-01-21T19:56:06.487 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-01-21T19:56:06.487 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-01-21T19:56:06.487 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-01-21T19:56:06.487 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-01-21T19:56:06.487 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-01-21T19:56:06.487 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-01-21T19:56:06.487 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-01-21T19:56:06.487 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-01-21T19:56:06.487 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-01-21T19:56:06.487 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-21T19:56:06.487 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-01-21T19:56:06.488 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-01-21T19:56:06.488 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-01-21T19:56:06.488 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-01-21T19:56:06.488 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-01-21T19:56:06.488 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-01-21T19:56:06.488 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-01-21T19:56:06.488 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-01-21T19:56:06.488 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-01-21T19:56:06.488 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-01-21T19:56:06.488 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-01-21T19:56:06.488 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-01-21T19:56:06.488 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-01-21T19:56:06.489 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-01-21T19:56:07.438 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-01-21T19:56:07.439 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:56:07.439 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-01-21T19:56:07.439 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-01-21T19:56:07.439 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-01-21T19:56:07.439 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-01-21T19:56:07.439 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-01-21T19:56:07.439 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-01-21T19:56:07.439 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-01-21T19:56:07.439 INFO:teuthology.orchestra.run.smithi053.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-01-21T19:56:07.439 INFO:teuthology.orchestra.run.smithi053.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-01-21T19:56:07.439 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-01-21T19:56:07.439 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-01-21T19:56:07.439 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-01-21T19:56:07.439 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-01-21T19:56:07.440 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-6.2-6.el8.noarch 2024-01-21T19:56:07.440 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-01-21T19:56:07.440 INFO:teuthology.orchestra.run.smithi053.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-01-21T19:56:07.440 INFO:teuthology.orchestra.run.smithi053.stdout: python3-mako-1.0.6-14.el8.noarch 2024-01-21T19:56:07.440 INFO:teuthology.orchestra.run.smithi053.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-01-21T19:56:07.440 INFO:teuthology.orchestra.run.smithi053.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-01-21T19:56:07.440 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-01-21T19:56:07.440 INFO:teuthology.orchestra.run.smithi053.stdout: python3-portend-2.6-1.el8.noarch 2024-01-21T19:56:07.440 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-01-21T19:56:07.440 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pycparser-2.14-14.el8.noarch 2024-01-21T19:56:07.440 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pytz-2017.2-11.el8.noarch 2024-01-21T19:56:07.440 INFO:teuthology.orchestra.run.smithi053.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-01-21T19:56:07.440 INFO:teuthology.orchestra.run.smithi053.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-01-21T19:56:07.440 INFO:teuthology.orchestra.run.smithi053.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-01-21T19:56:07.441 INFO:teuthology.orchestra.run.smithi053.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-01-21T19:56:07.441 INFO:teuthology.orchestra.run.smithi053.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-01-21T19:56:07.441 INFO:teuthology.orchestra.run.smithi053.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-01-21T19:56:07.441 INFO:teuthology.orchestra.run.smithi053.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-01-21T19:56:07.441 INFO:teuthology.orchestra.run.smithi053.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-01-21T19:56:07.441 INFO:teuthology.orchestra.run.smithi053.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-01-21T19:56:07.441 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:56:07.441 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-01-21T19:56:07.597 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-01-21T19:56:07.597 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:56:07.597 INFO:teuthology.orchestra.run.smithi146.stdout: Package Architecture Version Repository Size 2024-01-21T19:56:07.597 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:56:07.597 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-01-21T19:56:07.597 INFO:teuthology.orchestra.run.smithi146.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-01-21T19:56:07.598 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:56:07.598 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-01-21T19:56:07.598 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:56:07.598 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-01-21T19:56:07.598 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:56:07.598 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 71 k 2024-01-21T19:56:07.598 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 290 k 2024-01-21T19:56:07.599 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-01-21T19:56:07.647 INFO:teuthology.orchestra.run.smithi146.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.5 MB/s | 71 kB 00:00 2024-01-21T19:56:07.647 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:56:07.647 INFO:teuthology.orchestra.run.smithi146.stdout:Total 1.4 MB/s | 71 kB 00:00 2024-01-21T19:56:07.647 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-01-21T19:56:07.654 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-01-21T19:56:07.654 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-01-21T19:56:07.656 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-01-21T19:56:07.659 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-01-21T19:56:07.699 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-01-21T19:56:07.723 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-base 2024-01-21T19:56:08.215 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T19:56:08.432 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T19:56:08.485 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:14 ago on Sun 21 Jan 2024 07:54:54 PM UTC. 2024-01-21T19:56:08.659 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T19:56:09.020 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T19:56:09.021 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:56:09.021 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-01-21T19:56:09.021 INFO:teuthology.orchestra.run.smithi146.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-01-21T19:56:09.021 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:56:09.021 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-01-21T19:56:09.227 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-immutable-object-cache 2024-01-21T19:56:09.961 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:16 ago on Sun 21 Jan 2024 07:54:53 PM UTC. 2024-01-21T19:56:10.023 INFO:teuthology.orchestra.run.smithi053.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:56:10.073 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-01-21T19:56:10.074 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-01-21T19:56:10.075 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-01-21T19:56:10.182 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install cephadm 2024-01-21T19:56:10.938 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:16 ago on Sun 21 Jan 2024 07:54:54 PM UTC. 2024-01-21T19:56:11.442 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-01-21T19:56:11.443 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:56:11.443 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repo Size 2024-01-21T19:56:11.443 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:56:11.443 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-01-21T19:56:11.443 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-01-21T19:56:11.443 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:56:11.443 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-01-21T19:56:11.443 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:56:11.443 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-01-21T19:56:11.443 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:56:11.443 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 163 k 2024-01-21T19:56:11.443 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 587 k 2024-01-21T19:56:11.443 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-01-21T19:56:11.491 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 3.3 MB/s | 163 kB 00:00 2024-01-21T19:56:11.492 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:56:11.492 INFO:teuthology.orchestra.run.smithi146.stdout:Total 3.2 MB/s | 163 kB 00:00 2024-01-21T19:56:11.493 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-01-21T19:56:11.503 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-01-21T19:56:11.503 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-01-21T19:56:11.551 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-01-21T19:56:11.554 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-01-21T19:56:11.688 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-01-21T19:56:12.519 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-01-21T19:56:12.520 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:56:12.520 INFO:teuthology.orchestra.run.smithi053.stdout: Package Architecture Version Repository Size 2024-01-21T19:56:12.520 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:56:12.521 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-01-21T19:56:12.521 INFO:teuthology.orchestra.run.smithi053.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-01-21T19:56:12.521 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:56:12.521 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-01-21T19:56:12.521 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:56:12.521 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-01-21T19:56:12.521 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:56:12.521 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 71 k 2024-01-21T19:56:12.521 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 290 k 2024-01-21T19:56:12.521 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-01-21T19:56:12.569 INFO:teuthology.orchestra.run.smithi053.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.5 MB/s | 71 kB 00:00 2024-01-21T19:56:12.570 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:56:12.570 INFO:teuthology.orchestra.run.smithi053.stdout:Total 1.4 MB/s | 71 kB 00:00 2024-01-21T19:56:12.570 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-01-21T19:56:12.577 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-01-21T19:56:12.578 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-01-21T19:56:12.579 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-01-21T19:56:12.583 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-01-21T19:56:12.661 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-01-21T19:56:13.118 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T19:56:13.305 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T19:56:13.325 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:56:13.542 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T19:56:13.767 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:56:13.935 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T19:56:13.935 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:56:13.935 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-01-21T19:56:13.935 INFO:teuthology.orchestra.run.smithi053.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-01-21T19:56:13.935 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:56:13.935 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-01-21T19:56:14.164 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:56:14.164 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:56:14.164 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-01-21T19:56:14.164 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-01-21T19:56:14.164 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:56:14.164 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-01-21T19:56:14.196 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-immutable-object-cache 2024-01-21T19:56:14.390 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-mgr 2024-01-21T19:56:14.962 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:20 ago on Sun 21 Jan 2024 07:54:54 PM UTC. 2024-01-21T19:56:15.127 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:22 ago on Sun 21 Jan 2024 07:54:53 PM UTC. 2024-01-21T19:56:16.552 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-01-21T19:56:16.553 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:56:16.553 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repo Size 2024-01-21T19:56:16.553 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:56:16.554 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-01-21T19:56:16.554 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-01-21T19:56:16.554 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:56:16.554 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-01-21T19:56:16.554 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:56:16.554 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-01-21T19:56:16.554 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:56:16.554 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 163 k 2024-01-21T19:56:16.554 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 587 k 2024-01-21T19:56:16.554 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-01-21T19:56:16.561 INFO:teuthology.orchestra.run.smithi146.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:56:16.607 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-01-21T19:56:16.609 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-01-21T19:56:16.609 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-01-21T19:56:16.699 INFO:teuthology.orchestra.run.smithi053.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 1.1 MB/s | 163 kB 00:00 2024-01-21T19:56:16.700 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:56:16.700 INFO:teuthology.orchestra.run.smithi053.stdout:Total 1.1 MB/s | 163 kB 00:00 2024-01-21T19:56:16.700 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-01-21T19:56:16.712 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-01-21T19:56:16.712 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-01-21T19:56:16.721 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-mgr-dashboard 2024-01-21T19:56:16.763 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-01-21T19:56:16.766 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-01-21T19:56:16.903 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-01-21T19:56:17.208 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:56:17.441 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:24 ago on Sun 21 Jan 2024 07:54:53 PM UTC. 2024-01-21T19:56:17.643 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:56:18.052 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:56:18.052 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:56:18.052 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-01-21T19:56:18.052 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-01-21T19:56:18.052 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:56:18.052 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-01-21T19:56:18.296 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-mgr 2024-01-21T19:56:18.905 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-01-21T19:56:18.906 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:56:18.906 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-01-21T19:56:18.906 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:56:18.906 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-01-21T19:56:18.906 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-01-21T19:56:18.906 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-01-21T19:56:18.906 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-01-21T19:56:18.906 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-01-21T19:56:18.906 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-01-21T19:56:18.907 INFO:teuthology.orchestra.run.smithi146.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-01-21T19:56:18.907 INFO:teuthology.orchestra.run.smithi146.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-01-21T19:56:18.907 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:56:18.907 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-01-21T19:56:18.907 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:56:18.907 INFO:teuthology.orchestra.run.smithi146.stdout:Install 6 Packages 2024-01-21T19:56:18.907 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:56:18.907 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 3.0 M 2024-01-21T19:56:18.908 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 15 M 2024-01-21T19:56:18.908 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-01-21T19:56:19.057 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:25 ago on Sun 21 Jan 2024 07:54:54 PM UTC. 2024-01-21T19:56:19.212 INFO:teuthology.orchestra.run.smithi146.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 516 kB/s | 24 kB 00:00 2024-01-21T19:56:19.229 INFO:teuthology.orchestra.run.smithi146.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 141 kB/s | 8.9 kB 00:00 2024-01-21T19:56:19.313 INFO:teuthology.orchestra.run.smithi146.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 18 MB/s | 2.7 MB 00:00 2024-01-21T19:56:19.438 INFO:teuthology.orchestra.run.smithi146.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 189 kB/s | 43 kB 00:00 2024-01-21T19:56:19.613 INFO:teuthology.orchestra.run.smithi146.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 88 kB/s | 34 kB 00:00 2024-01-21T19:56:19.897 INFO:teuthology.orchestra.run.smithi146.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 339 kB/s | 196 kB 00:00 2024-01-21T19:56:19.900 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:56:19.900 INFO:teuthology.orchestra.run.smithi146.stdout:Total 3.0 MB/s | 3.0 MB 00:00 2024-01-21T19:56:19.910 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-01-21T19:56:19.920 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-01-21T19:56:19.921 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-01-21T19:56:20.104 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-01-21T19:56:20.107 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-01-21T19:56:20.424 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-01-21T19:56:20.601 INFO:teuthology.orchestra.run.smithi053.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:56:20.651 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-01-21T19:56:20.653 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-01-21T19:56:20.653 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-01-21T19:56:20.679 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-01-21T19:56:20.966 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-01-21T19:56:21.076 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-mgr-dashboard 2024-01-21T19:56:21.208 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-01-21T19:56:21.371 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-01-21T19:56:21.838 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:27 ago on Sun 21 Jan 2024 07:54:54 PM UTC. 2024-01-21T19:56:22.349 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-01-21T19:56:22.692 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-21T19:56:22.963 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-21T19:56:22.963 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-01-21T19:56:22.963 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-01-21T19:56:22.964 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-01-21T19:56:22.964 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-01-21T19:56:22.964 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-01-21T19:56:23.363 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-01-21T19:56:23.364 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:56:23.364 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-01-21T19:56:23.364 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-01-21T19:56:23.364 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-01-21T19:56:23.364 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-01-21T19:56:23.364 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-01-21T19:56:23.364 INFO:teuthology.orchestra.run.smithi146.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-01-21T19:56:23.364 INFO:teuthology.orchestra.run.smithi146.stdout: python3-routes-2.4.1-12.el8.noarch 2024-01-21T19:56:23.364 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:56:23.364 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-01-21T19:56:23.423 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-01-21T19:56:23.424 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:56:23.424 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-01-21T19:56:23.424 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:56:23.424 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-01-21T19:56:23.424 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-01-21T19:56:23.425 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-01-21T19:56:23.425 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-01-21T19:56:23.425 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-01-21T19:56:23.425 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-01-21T19:56:23.425 INFO:teuthology.orchestra.run.smithi053.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-01-21T19:56:23.425 INFO:teuthology.orchestra.run.smithi053.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-01-21T19:56:23.425 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:56:23.425 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-01-21T19:56:23.425 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:56:23.425 INFO:teuthology.orchestra.run.smithi053.stdout:Install 6 Packages 2024-01-21T19:56:23.425 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:56:23.426 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 3.0 M 2024-01-21T19:56:23.426 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 15 M 2024-01-21T19:56:23.426 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-01-21T19:56:23.538 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-21T19:56:23.552 INFO:teuthology.orchestra.run.smithi053.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 454 kB/s | 24 kB 00:00 2024-01-21T19:56:23.577 INFO:teuthology.orchestra.run.smithi053.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 114 kB/s | 8.9 kB 00:00 2024-01-21T19:56:23.644 INFO:teuthology.orchestra.run.smithi053.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 18 MB/s | 2.7 MB 00:00 2024-01-21T19:56:23.795 INFO:teuthology.orchestra.run.smithi053.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 176 kB/s | 43 kB 00:00 2024-01-21T19:56:23.953 INFO:teuthology.orchestra.run.smithi053.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 90 kB/s | 34 kB 00:00 2024-01-21T19:56:24.203 INFO:teuthology.orchestra.run.smithi053.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 352 kB/s | 196 kB 00:00 2024-01-21T19:56:24.207 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:56:24.207 INFO:teuthology.orchestra.run.smithi053.stdout:Total 3.8 MB/s | 3.0 MB 00:00 2024-01-21T19:56:24.218 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-01-21T19:56:24.229 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-01-21T19:56:24.229 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-01-21T19:56:24.267 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:31 ago on Sun 21 Jan 2024 07:54:53 PM UTC. 2024-01-21T19:56:24.406 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-01-21T19:56:24.409 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-01-21T19:56:24.737 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-01-21T19:56:24.949 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-01-21T19:56:25.177 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-01-21T19:56:25.368 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-01-21T19:56:25.523 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-01-21T19:56:25.755 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-01-21T19:56:25.757 INFO:teuthology.orchestra.run.smithi146.stdout:============================================================================================ 2024-01-21T19:56:25.757 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-01-21T19:56:25.757 INFO:teuthology.orchestra.run.smithi146.stdout:============================================================================================ 2024-01-21T19:56:25.757 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-01-21T19:56:25.757 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-01-21T19:56:25.757 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-01-21T19:56:25.757 INFO:teuthology.orchestra.run.smithi146.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-01-21T19:56:25.757 INFO:teuthology.orchestra.run.smithi146.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-01-21T19:56:25.757 INFO:teuthology.orchestra.run.smithi146.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-01-21T19:56:25.757 INFO:teuthology.orchestra.run.smithi146.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-01-21T19:56:25.757 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-01-21T19:56:25.757 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-01-21T19:56:25.757 INFO:teuthology.orchestra.run.smithi146.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-01-21T19:56:25.758 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:56:25.758 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-01-21T19:56:25.758 INFO:teuthology.orchestra.run.smithi146.stdout:============================================================================================ 2024-01-21T19:56:25.758 INFO:teuthology.orchestra.run.smithi146.stdout:Install 8 Packages 2024-01-21T19:56:25.758 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:56:25.758 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 37 M 2024-01-21T19:56:25.758 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 210 M 2024-01-21T19:56:25.759 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-01-21T19:56:26.144 INFO:teuthology.orchestra.run.smithi146.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 55 MB/s | 8.6 MB 00:00 2024-01-21T19:56:26.427 INFO:teuthology.orchestra.run.smithi146.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 11 MB/s | 4.9 MB 00:00 2024-01-21T19:56:26.532 INFO:teuthology.orchestra.run.smithi146.stdout:(3/8): openblas-0.3.15-4.el8.x86_64.rpm 8.7 MB/s | 4.7 MB 00:00 2024-01-21T19:56:26.563 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-01-21T19:56:26.569 INFO:teuthology.orchestra.run.smithi146.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.5 MB/s | 225 kB 00:00 2024-01-21T19:56:26.994 INFO:teuthology.orchestra.run.smithi146.stdout:(5/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.5 MB/s | 645 kB 00:00 2024-01-21T19:56:27.095 INFO:teuthology.orchestra.run.smithi146.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 1.7 MB/s | 172 kB 00:00 2024-01-21T19:56:27.260 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-21T19:56:27.278 INFO:teuthology.orchestra.run.smithi146.stdout:(7/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 3.3 MB/s | 3.7 MB 00:01 2024-01-21T19:56:27.593 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-21T19:56:27.593 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-01-21T19:56:27.593 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-01-21T19:56:27.593 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-01-21T19:56:27.593 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-01-21T19:56:27.593 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-01-21T19:56:27.997 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-01-21T19:56:27.998 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:56:27.998 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-01-21T19:56:27.998 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-01-21T19:56:27.998 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-01-21T19:56:27.998 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-01-21T19:56:27.998 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-01-21T19:56:27.998 INFO:teuthology.orchestra.run.smithi053.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-01-21T19:56:27.998 INFO:teuthology.orchestra.run.smithi053.stdout: python3-routes-2.4.1-12.el8.noarch 2024-01-21T19:56:27.998 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:56:27.998 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-01-21T19:56:28.127 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-21T19:56:28.237 INFO:teuthology.orchestra.run.smithi146.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 8.3 MB/s | 14 MB 00:01 2024-01-21T19:56:28.237 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:56:28.237 INFO:teuthology.orchestra.run.smithi146.stdout:Total 15 MB/s | 37 MB 00:02 2024-01-21T19:56:28.465 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-01-21T19:56:28.483 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-01-21T19:56:28.484 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-01-21T19:56:28.808 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-01-21T19:56:28.811 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-01-21T19:56:28.884 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:34 ago on Sun 21 Jan 2024 07:54:54 PM UTC. 2024-01-21T19:56:29.570 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-01-21T19:56:30.022 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-21T19:56:30.167 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-21T19:56:30.338 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-21T19:56:30.474 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-01-21T19:56:30.475 INFO:teuthology.orchestra.run.smithi053.stdout:============================================================================================ 2024-01-21T19:56:30.475 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-01-21T19:56:30.475 INFO:teuthology.orchestra.run.smithi053.stdout:============================================================================================ 2024-01-21T19:56:30.475 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-01-21T19:56:30.475 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-01-21T19:56:30.475 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-01-21T19:56:30.476 INFO:teuthology.orchestra.run.smithi053.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-01-21T19:56:30.476 INFO:teuthology.orchestra.run.smithi053.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-01-21T19:56:30.476 INFO:teuthology.orchestra.run.smithi053.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-01-21T19:56:30.476 INFO:teuthology.orchestra.run.smithi053.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-01-21T19:56:30.476 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-01-21T19:56:30.476 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-01-21T19:56:30.476 INFO:teuthology.orchestra.run.smithi053.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-01-21T19:56:30.476 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:56:30.476 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-01-21T19:56:30.476 INFO:teuthology.orchestra.run.smithi053.stdout:============================================================================================ 2024-01-21T19:56:30.476 INFO:teuthology.orchestra.run.smithi053.stdout:Install 8 Packages 2024-01-21T19:56:30.476 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:56:30.477 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 37 M 2024-01-21T19:56:30.477 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 210 M 2024-01-21T19:56:30.477 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-01-21T19:56:30.858 INFO:teuthology.orchestra.run.smithi053.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 58 MB/s | 8.6 MB 00:00 2024-01-21T19:56:30.940 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-21T19:56:31.172 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-01-21T19:56:31.399 INFO:teuthology.orchestra.run.smithi053.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 7.0 MB/s | 4.9 MB 00:00 2024-01-21T19:56:31.486 INFO:teuthology.orchestra.run.smithi053.stdout:(3/8): openblas-0.3.15-4.el8.x86_64.rpm 6.1 MB/s | 4.7 MB 00:00 2024-01-21T19:56:31.517 INFO:teuthology.orchestra.run.smithi053.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.9 MB/s | 225 kB 00:00 2024-01-21T19:56:31.584 INFO:teuthology.orchestra.run.smithi053.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 5.1 MB/s | 3.7 MB 00:00 2024-01-21T19:56:31.792 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-01-21T19:56:31.809 INFO:teuthology.orchestra.run.smithi053.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 767 kB/s | 172 kB 00:00 2024-01-21T19:56:31.834 INFO:teuthology.orchestra.run.smithi053.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 2.0 MB/s | 645 kB 00:00 2024-01-21T19:56:31.980 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-21T19:56:32.676 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-21T19:56:32.962 INFO:teuthology.orchestra.run.smithi053.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 9.6 MB/s | 14 MB 00:01 2024-01-21T19:56:32.962 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:56:32.963 INFO:teuthology.orchestra.run.smithi053.stdout:Total 15 MB/s | 37 MB 00:02 2024-01-21T19:56:33.041 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-01-21T19:56:33.204 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-01-21T19:56:33.223 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-01-21T19:56:33.224 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-01-21T19:56:33.305 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-21T19:56:33.570 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-01-21T19:56:33.573 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-01-21T19:56:34.230 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-01-21T19:56:34.398 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-21T19:56:34.533 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-21T19:56:34.689 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-21T19:56:35.082 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-21T19:56:35.312 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-21T19:56:35.506 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-01-21T19:56:36.141 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-01-21T19:56:36.331 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-21T19:56:36.710 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-01-21T19:56:36.959 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-21T19:56:37.047 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-21T19:56:37.255 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-21T19:56:37.256 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-01-21T19:56:37.256 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-01-21T19:56:37.256 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-01-21T19:56:37.256 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-01-21T19:56:37.256 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-01-21T19:56:37.256 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-01-21T19:56:37.256 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-01-21T19:56:37.389 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-01-21T19:56:37.656 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-01-21T19:56:37.656 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:56:37.657 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-01-21T19:56:37.657 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-01-21T19:56:37.657 INFO:teuthology.orchestra.run.smithi146.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-01-21T19:56:37.657 INFO:teuthology.orchestra.run.smithi146.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-01-21T19:56:37.657 INFO:teuthology.orchestra.run.smithi146.stdout: openblas-0.3.15-4.el8.x86_64 2024-01-21T19:56:37.657 INFO:teuthology.orchestra.run.smithi146.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-01-21T19:56:37.657 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-01-21T19:56:37.657 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-01-21T19:56:37.657 INFO:teuthology.orchestra.run.smithi146.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-01-21T19:56:37.657 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:56:37.657 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-01-21T19:56:37.781 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-mgr-rook 2024-01-21T19:56:38.439 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-21T19:56:38.511 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:45 ago on Sun 21 Jan 2024 07:54:53 PM UTC. 2024-01-21T19:56:40.018 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-01-21T19:56:40.019 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:56:40.019 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-01-21T19:56:40.020 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:56:40.020 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-01-21T19:56:40.020 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-01-21T19:56:40.020 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-01-21T19:56:40.020 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-01-21T19:56:40.020 INFO:teuthology.orchestra.run.smithi146.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-01-21T19:56:40.020 INFO:teuthology.orchestra.run.smithi146.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-01-21T19:56:40.020 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-01-21T19:56:40.020 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-01-21T19:56:40.020 INFO:teuthology.orchestra.run.smithi146.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-01-21T19:56:40.020 INFO:teuthology.orchestra.run.smithi146.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-01-21T19:56:40.020 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-01-21T19:56:40.021 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-01-21T19:56:40.021 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-01-21T19:56:40.021 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-01-21T19:56:40.021 INFO:teuthology.orchestra.run.smithi146.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-01-21T19:56:40.021 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:56:40.021 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-01-21T19:56:40.021 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:56:40.021 INFO:teuthology.orchestra.run.smithi146.stdout:Install 13 Packages 2024-01-21T19:56:40.021 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:56:40.022 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 1.9 M 2024-01-21T19:56:40.022 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 24 M 2024-01-21T19:56:40.022 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-01-21T19:56:40.337 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-21T19:56:40.461 INFO:teuthology.orchestra.run.smithi146.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 535 kB/s | 27 kB 00:00 2024-01-21T19:56:40.544 INFO:teuthology.orchestra.run.smithi146.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 199 kB/s | 27 kB 00:00 2024-01-21T19:56:40.570 INFO:teuthology.orchestra.run.smithi146.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 125 kB/s | 20 kB 00:00 2024-01-21T19:56:40.628 INFO:teuthology.orchestra.run.smithi146.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 1.3 MB/s | 110 kB 00:00 2024-01-21T19:56:40.653 INFO:teuthology.orchestra.run.smithi146.stdout:(5/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 657 kB/s | 126 kB 00:00 2024-01-21T19:56:40.796 INFO:teuthology.orchestra.run.smithi146.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 255 kB/s | 43 kB 00:00 2024-01-21T19:56:40.821 INFO:teuthology.orchestra.run.smithi146.stdout:(7/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 618 kB/s | 155 kB 00:00 2024-01-21T19:56:40.927 INFO:teuthology.orchestra.run.smithi146.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 120 kB/s | 33 kB 00:00 2024-01-21T19:56:41.347 INFO:teuthology.orchestra.run.smithi146.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 30 kB/s | 16 kB 00:00 2024-01-21T19:56:41.405 INFO:teuthology.orchestra.run.smithi146.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 139 kB/s | 81 kB 00:00 2024-01-21T19:56:41.464 INFO:teuthology.orchestra.run.smithi146.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 552 kB/s | 64 kB 00:00 2024-01-21T19:56:41.497 INFO:teuthology.orchestra.run.smithi146.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 670 kB/s | 61 kB 00:00 2024-01-21T19:56:41.539 INFO:teuthology.orchestra.run.smithi146.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 1.9 MB/s | 1.2 MB 00:00 2024-01-21T19:56:41.542 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:56:41.542 INFO:teuthology.orchestra.run.smithi146.stdout:Total 1.3 MB/s | 1.9 MB 00:01 2024-01-21T19:56:41.596 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-01-21T19:56:41.612 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-01-21T19:56:41.612 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-01-21T19:56:41.700 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-01-21T19:56:41.703 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-01-21T19:56:41.722 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-01-21T19:56:41.924 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-21T19:56:42.232 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-01-21T19:56:42.253 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-21T19:56:42.254 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-01-21T19:56:42.254 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-01-21T19:56:42.254 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-01-21T19:56:42.254 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-01-21T19:56:42.254 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-01-21T19:56:42.254 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-01-21T19:56:42.254 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-01-21T19:56:42.497 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-01-21T19:56:42.681 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-01-21T19:56:42.681 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:56:42.681 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-01-21T19:56:42.681 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-01-21T19:56:42.682 INFO:teuthology.orchestra.run.smithi053.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-01-21T19:56:42.682 INFO:teuthology.orchestra.run.smithi053.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-01-21T19:56:42.682 INFO:teuthology.orchestra.run.smithi053.stdout: openblas-0.3.15-4.el8.x86_64 2024-01-21T19:56:42.682 INFO:teuthology.orchestra.run.smithi053.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-01-21T19:56:42.682 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-01-21T19:56:42.682 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-01-21T19:56:42.682 INFO:teuthology.orchestra.run.smithi053.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-01-21T19:56:42.682 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:56:42.682 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-01-21T19:56:42.729 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-01-21T19:56:42.793 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-mgr-rook 2024-01-21T19:56:42.973 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-01-21T19:56:43.236 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-01-21T19:56:43.481 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-01-21T19:56:43.552 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:49 ago on Sun 21 Jan 2024 07:54:54 PM UTC. 2024-01-21T19:56:43.726 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-01-21T19:56:44.060 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-01-21T19:56:44.333 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-01-21T19:56:45.154 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-01-21T19:56:45.156 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:56:45.156 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-01-21T19:56:45.156 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:56:45.156 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-01-21T19:56:45.156 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-01-21T19:56:45.156 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-01-21T19:56:45.156 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-01-21T19:56:45.156 INFO:teuthology.orchestra.run.smithi053.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-01-21T19:56:45.157 INFO:teuthology.orchestra.run.smithi053.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-01-21T19:56:45.157 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-01-21T19:56:45.157 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-01-21T19:56:45.157 INFO:teuthology.orchestra.run.smithi053.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-01-21T19:56:45.157 INFO:teuthology.orchestra.run.smithi053.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-01-21T19:56:45.157 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-01-21T19:56:45.157 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-01-21T19:56:45.157 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-01-21T19:56:45.157 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-01-21T19:56:45.157 INFO:teuthology.orchestra.run.smithi053.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-01-21T19:56:45.157 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:56:45.157 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-01-21T19:56:45.157 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:56:45.157 INFO:teuthology.orchestra.run.smithi053.stdout:Install 13 Packages 2024-01-21T19:56:45.158 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:56:45.158 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 1.9 M 2024-01-21T19:56:45.158 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 24 M 2024-01-21T19:56:45.158 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-01-21T19:56:45.381 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-01-21T19:56:45.676 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-01-21T19:56:45.910 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-01-21T19:56:45.942 INFO:teuthology.orchestra.run.smithi053.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 354 kB/s | 27 kB 00:00 2024-01-21T19:56:46.034 INFO:teuthology.orchestra.run.smithi053.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 119 kB/s | 20 kB 00:00 2024-01-21T19:56:46.059 INFO:teuthology.orchestra.run.smithi053.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 138 kB/s | 27 kB 00:00 2024-01-21T19:56:46.118 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-01-21T19:56:46.268 INFO:teuthology.orchestra.run.smithi053.stdout:(4/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 744 kB/s | 155 kB 00:00 2024-01-21T19:56:46.310 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-21T19:56:46.327 INFO:teuthology.orchestra.run.smithi053.stdout:(5/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 329 kB/s | 126 kB 00:00 2024-01-21T19:56:46.352 INFO:teuthology.orchestra.run.smithi053.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 511 kB/s | 43 kB 00:00 2024-01-21T19:56:46.410 INFO:teuthology.orchestra.run.smithi053.stdout:(7/13): python3-pyasn1-modules-0.3.7-6.el8.noar 292 kB/s | 110 kB 00:00 2024-01-21T19:56:46.610 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-21T19:56:46.611 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-01-21T19:56:46.611 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-01-21T19:56:46.611 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-01-21T19:56:46.611 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-01-21T19:56:46.611 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-01-21T19:56:46.611 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-01-21T19:56:46.611 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-01-21T19:56:46.611 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-01-21T19:56:46.611 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-01-21T19:56:46.611 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-01-21T19:56:46.611 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-01-21T19:56:46.611 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-01-21T19:56:46.702 INFO:teuthology.orchestra.run.smithi053.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 46 kB/s | 16 kB 00:00 2024-01-21T19:56:46.769 INFO:teuthology.orchestra.run.smithi053.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 74 kB/s | 33 kB 00:00 2024-01-21T19:56:46.853 INFO:teuthology.orchestra.run.smithi053.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 184 kB/s | 81 kB 00:00 2024-01-21T19:56:46.978 INFO:teuthology.orchestra.run.smithi053.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 490 kB/s | 61 kB 00:00 2024-01-21T19:56:47.049 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-01-21T19:56:47.049 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:56:47.050 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-01-21T19:56:47.050 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-01-21T19:56:47.050 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-01-21T19:56:47.050 INFO:teuthology.orchestra.run.smithi146.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-01-21T19:56:47.050 INFO:teuthology.orchestra.run.smithi146.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-01-21T19:56:47.050 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-01-21T19:56:47.050 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-01-21T19:56:47.050 INFO:teuthology.orchestra.run.smithi146.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-01-21T19:56:47.050 INFO:teuthology.orchestra.run.smithi146.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-01-21T19:56:47.050 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-01-21T19:56:47.050 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-01-21T19:56:47.050 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-01-21T19:56:47.050 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rsa-4.9-2.el8.noarch 2024-01-21T19:56:47.050 INFO:teuthology.orchestra.run.smithi146.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-01-21T19:56:47.050 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:56:47.051 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-01-21T19:56:47.053 INFO:teuthology.orchestra.run.smithi053.stdout:(12/13): python3-rsa-4.9-2.el8.noarch.rpm 226 kB/s | 64 kB 00:00 2024-01-21T19:56:47.187 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-mgr-cephadm 2024-01-21T19:56:47.345 INFO:teuthology.orchestra.run.smithi053.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 1.8 MB/s | 1.2 MB 00:00 2024-01-21T19:56:47.346 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:56:47.346 INFO:teuthology.orchestra.run.smithi053.stdout:Total 894 kB/s | 1.9 MB 00:02 2024-01-21T19:56:47.406 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-01-21T19:56:47.423 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-01-21T19:56:47.423 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-01-21T19:56:47.518 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-01-21T19:56:47.521 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-01-21T19:56:47.912 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:54 ago on Sun 21 Jan 2024 07:54:53 PM UTC. 2024-01-21T19:56:48.090 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-01-21T19:56:48.334 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-01-21T19:56:48.534 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-01-21T19:56:48.817 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-01-21T19:56:49.003 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-01-21T19:56:49.234 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-01-21T19:56:49.394 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-01-21T19:56:49.395 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:56:49.395 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-01-21T19:56:49.396 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:56:49.396 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-01-21T19:56:49.396 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-01-21T19:56:49.396 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-01-21T19:56:49.396 INFO:teuthology.orchestra.run.smithi146.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-01-21T19:56:49.396 INFO:teuthology.orchestra.run.smithi146.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-01-21T19:56:49.396 INFO:teuthology.orchestra.run.smithi146.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-01-21T19:56:49.396 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-01-21T19:56:49.396 INFO:teuthology.orchestra.run.smithi146.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-01-21T19:56:49.396 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:56:49.396 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-01-21T19:56:49.396 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:56:49.396 INFO:teuthology.orchestra.run.smithi146.stdout:Install 6 Packages 2024-01-21T19:56:49.396 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:56:49.397 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 5.8 M 2024-01-21T19:56:49.397 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 24 M 2024-01-21T19:56:49.397 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-01-21T19:56:49.490 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-01-21T19:56:49.810 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-01-21T19:56:49.838 INFO:teuthology.orchestra.run.smithi146.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 2.2 MB/s | 89 kB 00:00 2024-01-21T19:56:49.981 INFO:teuthology.orchestra.run.smithi146.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 149 kB/s | 21 kB 00:00 2024-01-21T19:56:50.063 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-01-21T19:56:50.064 INFO:teuthology.orchestra.run.smithi146.stdout:(3/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 2.0 MB/s | 538 kB 00:00 2024-01-21T19:56:50.132 INFO:teuthology.orchestra.run.smithi146.stdout:(4/6): python3-execnet-1.7.1-1.el8.noarch.rpm 2.3 MB/s | 355 kB 00:00 2024-01-21T19:56:50.198 INFO:teuthology.orchestra.run.smithi146.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 366 kB/s | 49 kB 00:00 2024-01-21T19:56:50.465 INFO:teuthology.orchestra.run.smithi146.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 7.2 MB/s | 4.8 MB 00:00 2024-01-21T19:56:50.466 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:56:50.466 INFO:teuthology.orchestra.run.smithi146.stdout:Total 5.4 MB/s | 5.8 MB 00:01 2024-01-21T19:56:50.517 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-01-21T19:56:50.527 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-01-21T19:56:50.527 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-01-21T19:56:50.640 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-01-21T19:56:50.644 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-01-21T19:56:50.954 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-01-21T19:56:51.194 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-01-21T19:56:51.209 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-01-21T19:56:51.469 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-01-21T19:56:51.530 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-01-21T19:56:51.695 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-01-21T19:56:51.870 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-01-21T19:56:52.045 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-21T19:56:52.348 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-21T19:56:52.348 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-01-21T19:56:52.348 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-01-21T19:56:52.349 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-01-21T19:56:52.349 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-01-21T19:56:52.349 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-01-21T19:56:52.349 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-01-21T19:56:52.349 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-01-21T19:56:52.349 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-01-21T19:56:52.349 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-01-21T19:56:52.349 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-01-21T19:56:52.349 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-01-21T19:56:52.349 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-01-21T19:56:52.532 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-01-21T19:56:52.811 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-01-21T19:56:52.811 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:56:52.811 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-01-21T19:56:52.811 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-01-21T19:56:52.811 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-01-21T19:56:52.811 INFO:teuthology.orchestra.run.smithi053.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-01-21T19:56:52.811 INFO:teuthology.orchestra.run.smithi053.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-01-21T19:56:52.811 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-01-21T19:56:52.811 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-01-21T19:56:52.812 INFO:teuthology.orchestra.run.smithi053.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-01-21T19:56:52.812 INFO:teuthology.orchestra.run.smithi053.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-01-21T19:56:52.812 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-01-21T19:56:52.812 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-01-21T19:56:52.812 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-01-21T19:56:52.812 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rsa-4.9-2.el8.noarch 2024-01-21T19:56:52.812 INFO:teuthology.orchestra.run.smithi053.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-01-21T19:56:52.812 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:56:52.812 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-01-21T19:56:52.906 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-01-21T19:56:52.940 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-mgr-cephadm 2024-01-21T19:56:53.181 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2024-01-21T19:56:53.393 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-21T19:56:53.691 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-21T19:56:53.691 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-01-21T19:56:53.691 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-01-21T19:56:53.692 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2024-01-21T19:56:53.692 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-01-21T19:56:53.692 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-01-21T19:56:53.707 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:59 ago on Sun 21 Jan 2024 07:54:54 PM UTC. 2024-01-21T19:56:54.091 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-01-21T19:56:54.091 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:56:54.092 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-01-21T19:56:54.092 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-01-21T19:56:54.092 INFO:teuthology.orchestra.run.smithi146.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-01-21T19:56:54.092 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-01-21T19:56:54.092 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:56:54.092 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-01-21T19:56:54.260 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-fuse 2024-01-21T19:56:54.988 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:02:01 ago on Sun 21 Jan 2024 07:54:53 PM UTC. 2024-01-21T19:56:55.293 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-01-21T19:56:55.294 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:56:55.295 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-01-21T19:56:55.295 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:56:55.295 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-01-21T19:56:55.295 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-01-21T19:56:55.295 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-01-21T19:56:55.295 INFO:teuthology.orchestra.run.smithi053.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-01-21T19:56:55.295 INFO:teuthology.orchestra.run.smithi053.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-01-21T19:56:55.295 INFO:teuthology.orchestra.run.smithi053.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-01-21T19:56:55.295 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-01-21T19:56:55.295 INFO:teuthology.orchestra.run.smithi053.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-01-21T19:56:55.295 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:56:55.295 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-01-21T19:56:55.295 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:56:55.295 INFO:teuthology.orchestra.run.smithi053.stdout:Install 6 Packages 2024-01-21T19:56:55.296 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:56:55.296 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 5.8 M 2024-01-21T19:56:55.296 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 24 M 2024-01-21T19:56:55.296 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-01-21T19:56:55.436 INFO:teuthology.orchestra.run.smithi053.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 2.1 MB/s | 89 kB 00:00 2024-01-21T19:56:55.678 INFO:teuthology.orchestra.run.smithi053.stdout:(2/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.9 MB/s | 538 kB 00:00 2024-01-21T19:56:55.803 INFO:teuthology.orchestra.run.smithi053.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 58 kB/s | 21 kB 00:00 2024-01-21T19:56:55.987 INFO:teuthology.orchestra.run.smithi053.stdout:(4/6): python3-remoto-1.2.1-1.el8.noarch.rpm 267 kB/s | 49 kB 00:00 2024-01-21T19:56:56.337 INFO:teuthology.orchestra.run.smithi053.stdout:(5/6): python3-execnet-1.7.1-1.el8.noarch.rpm 539 kB/s | 355 kB 00:00 2024-01-21T19:56:56.468 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-01-21T19:56:56.469 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:56:56.469 INFO:teuthology.orchestra.run.smithi146.stdout: Package Architecture Version Repository Size 2024-01-21T19:56:56.470 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:56:56.470 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-01-21T19:56:56.470 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-01-21T19:56:56.470 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-01-21T19:56:56.470 INFO:teuthology.orchestra.run.smithi146.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-01-21T19:56:56.470 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:56:56.470 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-01-21T19:56:56.471 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:56:56.471 INFO:teuthology.orchestra.run.smithi146.stdout:Install 2 Packages 2024-01-21T19:56:56.471 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:56:56.471 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 968 k 2024-01-21T19:56:56.471 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 3.2 M 2024-01-21T19:56:56.471 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-01-21T19:56:56.529 INFO:teuthology.orchestra.run.smithi053.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 4.2 MB/s | 4.8 MB 00:01 2024-01-21T19:56:56.531 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:56:56.531 INFO:teuthology.orchestra.run.smithi053.stdout:Total 4.7 MB/s | 5.8 MB 00:01 2024-01-21T19:56:56.558 INFO:teuthology.orchestra.run.smithi146.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 19 MB/s | 885 kB 00:00 2024-01-21T19:56:56.587 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-01-21T19:56:56.598 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-01-21T19:56:56.598 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-01-21T19:56:56.711 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-01-21T19:56:56.714 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-01-21T19:56:56.716 INFO:teuthology.orchestra.run.smithi146.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 406 kB/s | 83 kB 00:00 2024-01-21T19:56:56.717 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:56:56.717 INFO:teuthology.orchestra.run.smithi146.stdout:Total 3.8 MB/s | 968 kB 00:00 2024-01-21T19:56:56.726 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-01-21T19:56:56.747 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-01-21T19:56:56.747 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-01-21T19:56:56.835 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-01-21T19:56:56.838 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-01-21T19:56:57.007 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-01-21T19:56:57.078 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-01-21T19:56:57.274 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-01-21T19:56:57.384 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-01-21T19:56:57.549 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-01-21T19:56:57.601 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-21T19:56:58.000 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-21T19:56:58.000 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-01-21T19:56:58.422 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-01-21T19:56:58.423 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:56:58.423 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-01-21T19:56:58.423 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-01-21T19:56:58.423 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:56:58.423 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-01-21T19:56:58.478 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-01-21T19:56:58.670 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install librados-devel 2024-01-21T19:56:58.847 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-01-21T19:56:59.411 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:02:06 ago on Sun 21 Jan 2024 07:54:53 PM UTC. 2024-01-21T19:56:59.495 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2024-01-21T19:56:59.869 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-21T19:57:00.222 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-21T19:57:00.222 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-01-21T19:57:00.222 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-01-21T19:57:00.222 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2024-01-21T19:57:00.222 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-01-21T19:57:00.222 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-01-21T19:57:00.654 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-01-21T19:57:00.654 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:57:00.654 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-01-21T19:57:00.654 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-01-21T19:57:00.654 INFO:teuthology.orchestra.run.smithi053.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-01-21T19:57:00.654 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-01-21T19:57:00.654 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:57:00.654 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-01-21T19:57:00.795 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-fuse 2024-01-21T19:57:00.896 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-01-21T19:57:00.896 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:57:00.896 INFO:teuthology.orchestra.run.smithi146.stdout: Package Architecture Version Repository Size 2024-01-21T19:57:00.897 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:57:00.897 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-01-21T19:57:00.897 INFO:teuthology.orchestra.run.smithi146.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-01-21T19:57:00.897 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:57:00.897 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-01-21T19:57:00.897 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:57:00.897 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-01-21T19:57:00.897 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:57:00.898 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 131 k 2024-01-21T19:57:00.898 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 439 k 2024-01-21T19:57:00.898 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-01-21T19:57:00.947 INFO:teuthology.orchestra.run.smithi146.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.6 MB/s | 131 kB 00:00 2024-01-21T19:57:00.947 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:57:00.948 INFO:teuthology.orchestra.run.smithi146.stdout:Total 2.6 MB/s | 131 kB 00:00 2024-01-21T19:57:00.948 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-01-21T19:57:00.953 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-01-21T19:57:00.953 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-01-21T19:57:01.003 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-01-21T19:57:01.006 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-01-21T19:57:01.111 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-01-21T19:57:01.316 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:57:01.554 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:07 ago on Sun 21 Jan 2024 07:54:54 PM UTC. 2024-01-21T19:57:01.643 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:57:02.572 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:57:02.572 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:57:02.572 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-01-21T19:57:02.572 INFO:teuthology.orchestra.run.smithi146.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-01-21T19:57:02.572 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:57:02.572 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-01-21T19:57:03.103 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install libcephfs2 2024-01-21T19:57:03.149 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-01-21T19:57:03.149 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:57:03.150 INFO:teuthology.orchestra.run.smithi053.stdout: Package Architecture Version Repository Size 2024-01-21T19:57:03.150 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:57:03.150 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-01-21T19:57:03.150 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-01-21T19:57:03.150 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-01-21T19:57:03.150 INFO:teuthology.orchestra.run.smithi053.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-01-21T19:57:03.150 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:57:03.150 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-01-21T19:57:03.150 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:57:03.150 INFO:teuthology.orchestra.run.smithi053.stdout:Install 2 Packages 2024-01-21T19:57:03.150 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:57:03.151 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 968 k 2024-01-21T19:57:03.151 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 3.2 M 2024-01-21T19:57:03.151 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-01-21T19:57:03.851 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:02:10 ago on Sun 21 Jan 2024 07:54:53 PM UTC. 2024-01-21T19:57:03.923 INFO:teuthology.orchestra.run.smithi053.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 11 MB/s | 885 kB 00:00 2024-01-21T19:57:04.198 INFO:teuthology.orchestra.run.smithi053.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 236 kB/s | 83 kB 00:00 2024-01-21T19:57:04.199 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:57:04.199 INFO:teuthology.orchestra.run.smithi053.stdout:Total 923 kB/s | 968 kB 00:01 2024-01-21T19:57:04.212 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-01-21T19:57:04.236 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-01-21T19:57:04.236 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-01-21T19:57:04.335 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-01-21T19:57:04.339 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-01-21T19:57:04.526 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-01-21T19:57:04.865 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-01-21T19:57:05.069 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-21T19:57:05.290 INFO:teuthology.orchestra.run.smithi146.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:57:05.335 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-01-21T19:57:05.336 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-01-21T19:57:05.337 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-01-21T19:57:05.437 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install libcephfs-devel 2024-01-21T19:57:05.479 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-21T19:57:05.479 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-01-21T19:57:05.875 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-01-21T19:57:05.876 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:57:05.876 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-01-21T19:57:05.876 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-01-21T19:57:05.876 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:57:05.876 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-01-21T19:57:06.090 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install librados-devel 2024-01-21T19:57:06.167 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:02:13 ago on Sun 21 Jan 2024 07:54:53 PM UTC. 2024-01-21T19:57:06.857 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:12 ago on Sun 21 Jan 2024 07:54:54 PM UTC. 2024-01-21T19:57:07.646 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-01-21T19:57:07.647 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:57:07.647 INFO:teuthology.orchestra.run.smithi146.stdout: Package Architecture Version Repository Size 2024-01-21T19:57:07.647 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:57:07.647 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-01-21T19:57:07.647 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-01-21T19:57:07.647 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:57:07.647 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-01-21T19:57:07.647 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:57:07.648 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-01-21T19:57:07.648 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:57:07.648 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 24 k 2024-01-21T19:57:07.648 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 94 k 2024-01-21T19:57:07.648 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-01-21T19:57:07.692 INFO:teuthology.orchestra.run.smithi146.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 556 kB/s | 24 kB 00:00 2024-01-21T19:57:07.692 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:57:07.693 INFO:teuthology.orchestra.run.smithi146.stdout:Total 541 kB/s | 24 kB 00:00 2024-01-21T19:57:07.693 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-01-21T19:57:07.696 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-01-21T19:57:07.696 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-01-21T19:57:07.699 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-01-21T19:57:07.702 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-01-21T19:57:07.776 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-01-21T19:57:07.964 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:57:08.419 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:57:08.447 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-01-21T19:57:08.447 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:57:08.447 INFO:teuthology.orchestra.run.smithi053.stdout: Package Architecture Version Repository Size 2024-01-21T19:57:08.447 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:57:08.448 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-01-21T19:57:08.448 INFO:teuthology.orchestra.run.smithi053.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-01-21T19:57:08.448 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:57:08.448 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-01-21T19:57:08.448 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:57:08.448 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-01-21T19:57:08.448 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:57:08.449 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 131 k 2024-01-21T19:57:08.449 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 439 k 2024-01-21T19:57:08.449 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-01-21T19:57:08.501 INFO:teuthology.orchestra.run.smithi053.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.4 MB/s | 131 kB 00:00 2024-01-21T19:57:08.502 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:57:08.502 INFO:teuthology.orchestra.run.smithi053.stdout:Total 2.4 MB/s | 131 kB 00:00 2024-01-21T19:57:08.502 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-01-21T19:57:08.508 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-01-21T19:57:08.508 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-01-21T19:57:08.561 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-01-21T19:57:08.564 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-01-21T19:57:08.676 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-01-21T19:57:08.909 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:57:09.232 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:57:09.253 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:57:09.253 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:57:09.253 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-01-21T19:57:09.253 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-01-21T19:57:09.254 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:57:09.254 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-01-21T19:57:09.377 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install librados2 2024-01-21T19:57:09.648 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:57:09.648 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:57:09.648 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-01-21T19:57:09.648 INFO:teuthology.orchestra.run.smithi053.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-01-21T19:57:09.648 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:57:09.649 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-01-21T19:57:10.116 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:02:17 ago on Sun 21 Jan 2024 07:54:53 PM UTC. 2024-01-21T19:57:10.952 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install libcephfs2 2024-01-21T19:57:11.545 INFO:teuthology.orchestra.run.smithi146.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:57:11.590 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-01-21T19:57:11.592 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-01-21T19:57:11.592 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-01-21T19:57:11.702 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install librbd1 2024-01-21T19:57:11.723 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:17 ago on Sun 21 Jan 2024 07:54:54 PM UTC. 2024-01-21T19:57:12.439 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:02:19 ago on Sun 21 Jan 2024 07:54:53 PM UTC. 2024-01-21T19:57:13.262 INFO:teuthology.orchestra.run.smithi053.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:57:13.311 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-01-21T19:57:13.312 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-01-21T19:57:13.312 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-01-21T19:57:13.418 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install libcephfs-devel 2024-01-21T19:57:13.879 INFO:teuthology.orchestra.run.smithi146.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:57:13.925 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-01-21T19:57:13.926 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-01-21T19:57:13.926 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-01-21T19:57:14.033 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install python3-rados 2024-01-21T19:57:14.174 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:20 ago on Sun 21 Jan 2024 07:54:54 PM UTC. 2024-01-21T19:57:14.764 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:02:21 ago on Sun 21 Jan 2024 07:54:53 PM UTC. 2024-01-21T19:57:15.773 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-01-21T19:57:15.773 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:57:15.773 INFO:teuthology.orchestra.run.smithi053.stdout: Package Architecture Version Repository Size 2024-01-21T19:57:15.773 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:57:15.773 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-01-21T19:57:15.774 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-01-21T19:57:15.774 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:57:15.774 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-01-21T19:57:15.774 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:57:15.774 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-01-21T19:57:15.774 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:57:15.775 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 24 k 2024-01-21T19:57:15.775 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 94 k 2024-01-21T19:57:15.775 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-01-21T19:57:15.833 INFO:teuthology.orchestra.run.smithi053.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 404 kB/s | 24 kB 00:00 2024-01-21T19:57:15.834 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:57:15.834 INFO:teuthology.orchestra.run.smithi053.stdout:Total 396 kB/s | 24 kB 00:00 2024-01-21T19:57:15.834 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-01-21T19:57:15.838 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-01-21T19:57:15.838 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-01-21T19:57:15.840 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-01-21T19:57:15.844 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-01-21T19:57:15.898 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-01-21T19:57:16.124 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:57:16.194 INFO:teuthology.orchestra.run.smithi146.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:57:16.239 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-01-21T19:57:16.241 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-01-21T19:57:16.241 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-01-21T19:57:16.345 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install python3-rgw 2024-01-21T19:57:16.396 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:57:16.775 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:57:16.775 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:57:16.775 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-01-21T19:57:16.775 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-01-21T19:57:16.775 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:57:16.776 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-01-21T19:57:16.910 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install librados2 2024-01-21T19:57:17.077 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:02:24 ago on Sun 21 Jan 2024 07:54:53 PM UTC. 2024-01-21T19:57:17.668 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:23 ago on Sun 21 Jan 2024 07:54:54 PM UTC. 2024-01-21T19:57:18.502 INFO:teuthology.orchestra.run.smithi146.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:57:18.547 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-01-21T19:57:18.548 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-01-21T19:57:18.548 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-01-21T19:57:18.657 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install python3-cephfs 2024-01-21T19:57:19.204 INFO:teuthology.orchestra.run.smithi053.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:57:19.253 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-01-21T19:57:19.254 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-01-21T19:57:19.254 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-01-21T19:57:19.363 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install librbd1 2024-01-21T19:57:19.380 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:02:26 ago on Sun 21 Jan 2024 07:54:53 PM UTC. 2024-01-21T19:57:20.117 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:26 ago on Sun 21 Jan 2024 07:54:54 PM UTC. 2024-01-21T19:57:20.860 INFO:teuthology.orchestra.run.smithi146.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:57:20.906 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-01-21T19:57:20.907 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-01-21T19:57:20.907 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-01-21T19:57:21.016 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install python3-rbd 2024-01-21T19:57:21.662 INFO:teuthology.orchestra.run.smithi053.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:57:21.711 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-01-21T19:57:21.712 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-01-21T19:57:21.712 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-01-21T19:57:21.740 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:02:28 ago on Sun 21 Jan 2024 07:54:53 PM UTC. 2024-01-21T19:57:21.835 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install python3-rados 2024-01-21T19:57:22.620 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:28 ago on Sun 21 Jan 2024 07:54:54 PM UTC. 2024-01-21T19:57:23.187 INFO:teuthology.orchestra.run.smithi146.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:57:23.233 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-01-21T19:57:23.234 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-01-21T19:57:23.234 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-01-21T19:57:23.336 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install rbd-fuse 2024-01-21T19:57:24.060 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:02:31 ago on Sun 21 Jan 2024 07:54:53 PM UTC. 2024-01-21T19:57:24.156 INFO:teuthology.orchestra.run.smithi053.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:57:24.204 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-01-21T19:57:24.206 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-01-21T19:57:24.206 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-01-21T19:57:24.341 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install python3-rgw 2024-01-21T19:57:25.102 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:31 ago on Sun 21 Jan 2024 07:54:54 PM UTC. 2024-01-21T19:57:25.529 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-01-21T19:57:25.530 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:57:25.530 INFO:teuthology.orchestra.run.smithi146.stdout: Package Architecture Version Repository Size 2024-01-21T19:57:25.530 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:57:25.530 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-01-21T19:57:25.530 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-01-21T19:57:25.530 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:57:25.531 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-01-21T19:57:25.531 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:57:25.531 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-01-21T19:57:25.531 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:57:25.531 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 89 k 2024-01-21T19:57:25.531 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 298 k 2024-01-21T19:57:25.531 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-01-21T19:57:25.576 INFO:teuthology.orchestra.run.smithi146.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 2.0 MB/s | 89 kB 00:00 2024-01-21T19:57:25.576 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:57:25.577 INFO:teuthology.orchestra.run.smithi146.stdout:Total 1.9 MB/s | 89 kB 00:00 2024-01-21T19:57:25.577 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-01-21T19:57:25.586 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-01-21T19:57:25.586 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-01-21T19:57:25.636 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-01-21T19:57:25.639 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-01-21T19:57:25.776 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-01-21T19:57:25.987 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:57:26.264 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:57:26.631 INFO:teuthology.orchestra.run.smithi053.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:57:26.660 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:57:26.660 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:57:26.660 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-01-21T19:57:26.661 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-01-21T19:57:26.661 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:57:26.661 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-01-21T19:57:26.681 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-01-21T19:57:26.682 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-01-21T19:57:26.682 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-01-21T19:57:26.799 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install python3-cephfs 2024-01-21T19:57:26.906 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install rbd-mirror 2024-01-21T19:57:27.556 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:33 ago on Sun 21 Jan 2024 07:54:54 PM UTC. 2024-01-21T19:57:27.647 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:02:34 ago on Sun 21 Jan 2024 07:54:53 PM UTC. 2024-01-21T19:57:29.093 INFO:teuthology.orchestra.run.smithi053.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:57:29.143 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-01-21T19:57:29.144 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-01-21T19:57:29.144 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-01-21T19:57:29.145 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-01-21T19:57:29.146 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:57:29.146 INFO:teuthology.orchestra.run.smithi146.stdout: Package Architecture Version Repository Size 2024-01-21T19:57:29.146 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:57:29.147 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-01-21T19:57:29.147 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-21T19:57:29.147 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:57:29.147 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-01-21T19:57:29.147 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:57:29.147 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-01-21T19:57:29.147 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:57:29.148 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 3.7 M 2024-01-21T19:57:29.148 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 18 M 2024-01-21T19:57:29.148 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-01-21T19:57:29.237 INFO:teuthology.orchestra.run.smithi146.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 42 MB/s | 3.7 MB 00:00 2024-01-21T19:57:29.239 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:57:29.239 INFO:teuthology.orchestra.run.smithi146.stdout:Total 41 MB/s | 3.7 MB 00:00 2024-01-21T19:57:29.239 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-01-21T19:57:29.250 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-01-21T19:57:29.250 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-01-21T19:57:29.251 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install python3-rbd 2024-01-21T19:57:29.314 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-01-21T19:57:29.317 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-01-21T19:57:29.787 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-01-21T19:57:30.015 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:36 ago on Sun 21 Jan 2024 07:54:54 PM UTC. 2024-01-21T19:57:30.021 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:57:30.462 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:57:30.827 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:57:30.827 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:57:30.827 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-01-21T19:57:30.827 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-01-21T19:57:30.827 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:57:30.827 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-01-21T19:57:31.054 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install rbd-nbd 2024-01-21T19:57:31.549 INFO:teuthology.orchestra.run.smithi053.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:57:31.597 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-01-21T19:57:31.599 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-01-21T19:57:31.599 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-01-21T19:57:31.700 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install rbd-fuse 2024-01-21T19:57:31.798 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:02:38 ago on Sun 21 Jan 2024 07:54:53 PM UTC. 2024-01-21T19:57:32.466 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:38 ago on Sun 21 Jan 2024 07:54:54 PM UTC. 2024-01-21T19:57:33.289 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-01-21T19:57:33.290 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:57:33.290 INFO:teuthology.orchestra.run.smithi146.stdout: Package Architecture Version Repository Size 2024-01-21T19:57:33.290 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:57:33.290 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-01-21T19:57:33.290 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-01-21T19:57:33.290 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:57:33.290 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-01-21T19:57:33.290 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-21T19:57:33.290 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-01-21T19:57:33.290 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:57:33.290 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 173 k 2024-01-21T19:57:33.290 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 649 k 2024-01-21T19:57:33.291 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-01-21T19:57:33.340 INFO:teuthology.orchestra.run.smithi146.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.4 MB/s | 173 kB 00:00 2024-01-21T19:57:33.341 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:57:33.341 INFO:teuthology.orchestra.run.smithi146.stdout:Total 3.3 MB/s | 173 kB 00:00 2024-01-21T19:57:33.341 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-01-21T19:57:33.352 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-01-21T19:57:33.352 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-01-21T19:57:33.402 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-01-21T19:57:33.405 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-01-21T19:57:33.518 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-01-21T19:57:33.702 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:57:34.018 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:57:34.054 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-01-21T19:57:34.054 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:57:34.054 INFO:teuthology.orchestra.run.smithi053.stdout: Package Architecture Version Repository Size 2024-01-21T19:57:34.055 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:57:34.055 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-01-21T19:57:34.055 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-01-21T19:57:34.055 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:57:34.055 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-01-21T19:57:34.055 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:57:34.055 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-01-21T19:57:34.055 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:57:34.056 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 89 k 2024-01-21T19:57:34.056 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 298 k 2024-01-21T19:57:34.056 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-01-21T19:57:34.106 INFO:teuthology.orchestra.run.smithi053.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.7 MB/s | 89 kB 00:00 2024-01-21T19:57:34.107 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:57:34.107 INFO:teuthology.orchestra.run.smithi053.stdout:Total 1.7 MB/s | 89 kB 00:00 2024-01-21T19:57:34.107 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-01-21T19:57:34.118 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-01-21T19:57:34.118 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-01-21T19:57:34.170 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-01-21T19:57:34.173 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-01-21T19:57:34.285 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-01-21T19:57:34.408 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:57:34.409 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:57:34.409 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-01-21T19:57:34.409 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-01-21T19:57:34.409 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:57:34.409 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-01-21T19:57:34.630 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:57:34.636 DEBUG:teuthology.parallel:result is None 2024-01-21T19:57:34.936 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:57:35.332 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:57:35.332 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:57:35.332 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-01-21T19:57:35.332 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-01-21T19:57:35.332 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:57:35.332 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-01-21T19:57:35.527 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install rbd-mirror 2024-01-21T19:57:36.292 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:42 ago on Sun 21 Jan 2024 07:54:54 PM UTC. 2024-01-21T19:57:37.883 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-01-21T19:57:37.884 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:57:37.884 INFO:teuthology.orchestra.run.smithi053.stdout: Package Architecture Version Repository Size 2024-01-21T19:57:37.884 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:57:37.884 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-01-21T19:57:37.884 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-21T19:57:37.884 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:57:37.884 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-01-21T19:57:37.884 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:57:37.885 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-01-21T19:57:37.885 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:57:37.885 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 3.7 M 2024-01-21T19:57:37.885 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 18 M 2024-01-21T19:57:37.885 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-01-21T19:57:37.984 INFO:teuthology.orchestra.run.smithi053.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 37 MB/s | 3.7 MB 00:00 2024-01-21T19:57:37.985 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:57:37.985 INFO:teuthology.orchestra.run.smithi053.stdout:Total 37 MB/s | 3.7 MB 00:00 2024-01-21T19:57:37.986 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-01-21T19:57:37.999 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-01-21T19:57:37.999 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-01-21T19:57:38.066 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-01-21T19:57:38.069 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-01-21T19:57:38.693 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-01-21T19:57:38.981 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:57:39.439 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:57:39.887 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:57:39.887 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:57:39.887 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-01-21T19:57:39.887 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-01-21T19:57:39.887 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:57:39.887 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-01-21T19:57:40.122 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install rbd-nbd 2024-01-21T19:57:40.885 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:46 ago on Sun 21 Jan 2024 07:54:54 PM UTC. 2024-01-21T19:57:42.490 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-01-21T19:57:42.490 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:57:42.490 INFO:teuthology.orchestra.run.smithi053.stdout: Package Architecture Version Repository Size 2024-01-21T19:57:42.491 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:57:42.491 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-01-21T19:57:42.491 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-01-21T19:57:42.491 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:57:42.491 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-01-21T19:57:42.491 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-21T19:57:42.491 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-01-21T19:57:42.491 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:57:42.492 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 173 k 2024-01-21T19:57:42.492 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 649 k 2024-01-21T19:57:42.492 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-01-21T19:57:42.556 INFO:teuthology.orchestra.run.smithi053.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 2.6 MB/s | 173 kB 00:00 2024-01-21T19:57:42.556 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:57:42.557 INFO:teuthology.orchestra.run.smithi053.stdout:Total 2.6 MB/s | 173 kB 00:00 2024-01-21T19:57:42.557 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-01-21T19:57:42.569 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-01-21T19:57:42.569 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-01-21T19:57:42.623 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-01-21T19:57:42.627 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-01-21T19:57:42.745 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-01-21T19:57:42.929 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:57:43.265 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:57:43.686 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:57:43.686 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:57:43.686 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-01-21T19:57:43.687 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-01-21T19:57:43.687 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-21T19:57:43.687 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-01-21T19:57:43.931 DEBUG:teuthology.parallel:result is None 2024-01-21T19:57:43.931 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-21T19:57:43.932 INFO:teuthology.packaging:ref: None 2024-01-21T19:57:43.932 INFO:teuthology.packaging:tag: v16.2.4 2024-01-21T19:57:43.932 INFO:teuthology.packaging:branch: None 2024-01-21T19:57:43.932 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T19:57:43.932 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-21T19:57:44.076 DEBUG:teuthology.orchestra.run.smithi053:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-21T19:57:44.099 INFO:teuthology.orchestra.run.smithi053.stdout:16.2.4-0.el8 2024-01-21T19:57:44.100 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-01-21T19:57:44.100 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-01-21T19:57:44.102 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-21T19:57:44.102 INFO:teuthology.packaging:ref: None 2024-01-21T19:57:44.102 INFO:teuthology.packaging:tag: v16.2.4 2024-01-21T19:57:44.102 INFO:teuthology.packaging:branch: None 2024-01-21T19:57:44.102 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T19:57:44.102 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-21T19:57:44.242 DEBUG:teuthology.orchestra.run.smithi146:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-21T19:57:44.269 INFO:teuthology.orchestra.run.smithi146.stdout:16.2.4-0.el8 2024-01-21T19:57:44.270 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-01-21T19:57:44.270 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-01-21T19:57:44.272 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-01-21T19:57:44.272 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-21T19:57:44.272 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-21T19:57:44.301 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-21T19:57:44.302 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-21T19:57:44.341 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-01-21T19:57:44.341 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-21T19:57:44.341 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/daemon-helper 2024-01-21T19:57:44.369 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-21T19:57:44.437 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-21T19:57:44.437 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/daemon-helper 2024-01-21T19:57:44.467 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-21T19:57:44.536 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-01-21T19:57:44.536 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-21T19:57:44.536 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-21T19:57:44.564 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-21T19:57:44.632 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-21T19:57:44.632 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-21T19:57:44.660 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-21T19:57:44.729 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-01-21T19:57:44.729 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-21T19:57:44.729 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/stdin-killer 2024-01-21T19:57:44.757 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-21T19:57:44.825 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-21T19:57:44.825 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/stdin-killer 2024-01-21T19:57:44.853 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-21T19:57:44.921 INFO:teuthology.run_tasks:Running task print... 2024-01-21T19:57:44.933 INFO:teuthology.task.print:**** done install task... 2024-01-21T19:57:44.933 INFO:teuthology.run_tasks:Running task cephadm... 2024-01-21T19:57:45.039 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': 'ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec'} 2024-01-21T19:57:45.040 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2024-01-21T19:57:45.040 INFO:tasks.cephadm:Cluster fsid is 57f6722c-b897-11ee-95b0-87774f69a715 2024-01-21T19:57:45.040 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-01-21T19:57:45.040 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-01-21T19:57:45.041 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi053': '172.21.15.53', 'mon.smithi146': '172.21.15.146'} 2024-01-21T19:57:45.041 INFO:tasks.cephadm:Normalizing hostnames... 2024-01-21T19:57:45.041 DEBUG:teuthology.orchestra.run.smithi053:> sudo hostname $(hostname -s) 2024-01-21T19:57:45.069 DEBUG:teuthology.orchestra.run.smithi146:> sudo hostname $(hostname -s) 2024-01-21T19:57:45.097 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2024-01-21T19:57:45.098 DEBUG:teuthology.orchestra.run.smithi053:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-21T19:57:45.358 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Jan 21 19:57 /home/ubuntu/cephtest/cephadm 2024-01-21T19:57:45.359 DEBUG:teuthology.orchestra.run.smithi146:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-21T19:57:45.510 INFO:teuthology.orchestra.run.smithi146.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Jan 21 19:57 /home/ubuntu/cephtest/cephadm 2024-01-21T19:57:45.510 DEBUG:teuthology.orchestra.run.smithi053:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-21T19:57:45.533 DEBUG:teuthology.orchestra.run.smithi146:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-21T19:57:45.563 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /etc/ceph 2024-01-21T19:57:45.606 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /etc/ceph 2024-01-21T19:57:45.633 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 777 /etc/ceph 2024-01-21T19:57:45.675 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 777 /etc/ceph 2024-01-21T19:57:45.702 INFO:tasks.cephadm:Writing seed config... 2024-01-21T19:57:45.703 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-01-21T19:57:45.703 INFO:tasks.cephadm: override: [client] debug client = 20 2024-01-21T19:57:45.704 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-01-21T19:57:45.704 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-01-21T19:57:45.704 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-01-21T19:57:45.704 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-01-21T19:57:45.704 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-01-21T19:57:45.704 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-01-21T19:57:45.704 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-01-21T19:57:45.704 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-01-21T19:57:45.704 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-01-21T19:57:45.704 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-01-21T19:57:45.704 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-01-21T19:57:45.704 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-01-21T19:57:45.704 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-01-21T19:57:45.704 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-01-21T19:57:45.705 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-01-21T19:57:45.705 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-01-21T19:57:45.705 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-01-21T19:57:45.705 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-01-21T19:57:45.705 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-01-21T19:57:45.705 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-01-21T19:57:45.705 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-01-21T19:57:45.705 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-01-21T19:57:45.705 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-01-21T19:57:45.705 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-01-21T19:57:45.705 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-01-21T19:57:45.705 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-01-21T19:57:45.705 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-01-21T19:57:45.705 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-01-21T19:57:45.705 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-01-21T19:57:45.706 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-01-21T19:57:45.706 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-01-21T19:57:45.706 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-01-21T19:57:45.706 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-01-21T19:57:45.706 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-01-21T19:57:45.706 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-01-21T19:57:45.706 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-01-21T19:57:45.706 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-21T19:57:45.707 DEBUG:teuthology.orchestra.run.smithi053:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-01-21T19:57:45.735 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 57f6722c-b897-11ee-95b0-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-01-21T19:57:45.735 DEBUG:teuthology.orchestra.run.smithi053:mon.smithi053> sudo journalctl -f -n 0 -u ceph-57f6722c-b897-11ee-95b0-87774f69a715@mon.smithi053.service 2024-01-21T19:57:45.778 INFO:tasks.cephadm:Bootstrapping... 2024-01-21T19:57:45.778 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid 57f6722c-b897-11ee-95b0-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.53 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-01-21T19:57:45.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:-- Logs begin at Sun 2024-01-21 19:47:17 UTC. -- 2024-01-21T19:57:46.062 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-01-21T19:57:47.024 INFO:teuthology.orchestra.run.smithi053.stderr:/bin/podman: 4.6.1 2024-01-21T19:57:47.034 INFO:teuthology.orchestra.run.smithi053.stderr:Verifying podman|docker is present... 2024-01-21T19:57:47.035 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-01-21T19:57:47.058 INFO:teuthology.orchestra.run.smithi053.stderr:/bin/podman: 4.6.1 2024-01-21T19:57:47.068 INFO:teuthology.orchestra.run.smithi053.stderr:Verifying lvm2 is present... 2024-01-21T19:57:47.068 INFO:teuthology.orchestra.run.smithi053.stderr:Verifying time synchronization is in place... 2024-01-21T19:57:47.068 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl is-enabled chrony.service 2024-01-21T19:57:47.076 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-01-21T19:57:47.077 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl is-active chrony.service 2024-01-21T19:57:47.083 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: inactive 2024-01-21T19:57:47.083 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl is-enabled chronyd.service 2024-01-21T19:57:47.090 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: enabled 2024-01-21T19:57:47.090 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl is-active chronyd.service 2024-01-21T19:57:47.096 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: active 2024-01-21T19:57:47.097 INFO:teuthology.orchestra.run.smithi053.stderr:Unit chronyd.service is enabled and running 2024-01-21T19:57:47.097 INFO:teuthology.orchestra.run.smithi053.stderr:Repeating the final host check... 2024-01-21T19:57:47.097 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-01-21T19:57:47.118 INFO:teuthology.orchestra.run.smithi053.stderr:/bin/podman: 4.6.1 2024-01-21T19:57:47.131 INFO:teuthology.orchestra.run.smithi053.stderr:podman|docker (/bin/podman) is present 2024-01-21T19:57:47.132 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl is present 2024-01-21T19:57:47.132 INFO:teuthology.orchestra.run.smithi053.stderr:lvcreate is present 2024-01-21T19:57:47.132 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl is-enabled chrony.service 2024-01-21T19:57:47.137 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-01-21T19:57:47.138 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl is-active chrony.service 2024-01-21T19:57:47.143 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: inactive 2024-01-21T19:57:47.144 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl is-enabled chronyd.service 2024-01-21T19:57:47.150 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: enabled 2024-01-21T19:57:47.150 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl is-active chronyd.service 2024-01-21T19:57:47.155 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: active 2024-01-21T19:57:47.156 INFO:teuthology.orchestra.run.smithi053.stderr:Unit chronyd.service is enabled and running 2024-01-21T19:57:47.156 INFO:teuthology.orchestra.run.smithi053.stderr:Host looks OK 2024-01-21T19:57:47.156 INFO:teuthology.orchestra.run.smithi053.stderr:Cluster fsid: 57f6722c-b897-11ee-95b0-87774f69a715 2024-01-21T19:57:47.157 INFO:teuthology.orchestra.run.smithi053.stderr:Acquiring lock 140640071911912 on /run/cephadm/57f6722c-b897-11ee-95b0-87774f69a715.lock 2024-01-21T19:57:47.157 INFO:teuthology.orchestra.run.smithi053.stderr:Lock 140640071911912 acquired on /run/cephadm/57f6722c-b897-11ee-95b0-87774f69a715.lock 2024-01-21T19:57:47.157 INFO:teuthology.orchestra.run.smithi053.stderr:Verifying IP 172.21.15.53 port 3300 ... 2024-01-21T19:57:47.157 INFO:teuthology.orchestra.run.smithi053.stderr:Verifying IP 172.21.15.53 port 6789 ... 2024-01-21T19:57:47.158 INFO:teuthology.orchestra.run.smithi053.stderr:Base mon IP is 172.21.15.53, final addrv is [v2:172.21.15.53:3300,v1:172.21.15.53:6789] 2024-01-21T19:57:47.158 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /sbin/ip route ls 2024-01-21T19:57:47.160 INFO:teuthology.orchestra.run.smithi053.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.53 metric 100 2024-01-21T19:57:47.160 INFO:teuthology.orchestra.run.smithi053.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.53 metric 100 2024-01-21T19:57:47.161 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /sbin/ip -6 route ls 2024-01-21T19:57:47.163 INFO:teuthology.orchestra.run.smithi053.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-01-21T19:57:47.163 INFO:teuthology.orchestra.run.smithi053.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2024-01-21T19:57:47.164 INFO:teuthology.orchestra.run.smithi053.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1734sec hoplimit 64 pref medium 2024-01-21T19:57:47.164 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /sbin/ip -6 addr ls 2024-01-21T19:57:47.166 INFO:teuthology.orchestra.run.smithi053.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-01-21T19:57:47.167 INFO:teuthology.orchestra.run.smithi053.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-01-21T19:57:47.167 INFO:teuthology.orchestra.run.smithi053.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-01-21T19:57:47.167 INFO:teuthology.orchestra.run.smithi053.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2024-01-21T19:57:47.167 INFO:teuthology.orchestra.run.smithi053.stderr:/sbin/ip: inet6 fe80::ae1f:6bff:fef8:3718/64 scope link 2024-01-21T19:57:47.167 INFO:teuthology.orchestra.run.smithi053.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-01-21T19:57:47.168 INFO:teuthology.orchestra.run.smithi053.stderr:Mon IP 172.21.15.53 is in CIDR network 172.21.0.0/20 2024-01-21T19:57:47.168 INFO:teuthology.orchestra.run.smithi053.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-01-21T19:57:47.169 INFO:teuthology.orchestra.run.smithi053.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2024-01-21T19:57:47.169 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2024-01-21T19:57:47.273 INFO:teuthology.orchestra.run.smithi053.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2024-01-21T19:57:53.933 INFO:teuthology.orchestra.run.smithi053.stderr:/bin/podman: Getting image source signatures 2024-01-21T19:57:53.946 INFO:teuthology.orchestra.run.smithi053.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2024-01-21T19:57:53.946 INFO:teuthology.orchestra.run.smithi053.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2024-01-21T19:58:06.825 INFO:teuthology.orchestra.run.smithi053.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-01-21T19:58:06.874 INFO:teuthology.orchestra.run.smithi053.stderr:/bin/podman: Writing manifest to image destination 2024-01-21T19:58:07.193 INFO:teuthology.orchestra.run.smithi053.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-01-21T19:58:07.206 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2024-01-21T19:58:07.971 INFO:teuthology.orchestra.run.smithi053.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-01-21T19:58:08.324 INFO:teuthology.orchestra.run.smithi053.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-01-21T19:58:08.324 INFO:teuthology.orchestra.run.smithi053.stderr:Extracting ceph user uid/gid from container image... 2024-01-21T19:58:08.324 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2024-01-21T19:58:08.602 INFO:teuthology.orchestra.run.smithi053.stderr:stat: 167 167 2024-01-21T19:58:08.927 INFO:teuthology.orchestra.run.smithi053.stderr:Creating initial keys... 2024-01-21T19:58:08.927 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-01-21T19:58:09.227 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-authtool: AQBRd61lwGnPDRAABN2Y9tfjZayHzVmZk63L3g== 2024-01-21T19:58:09.585 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-01-21T19:58:09.910 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-authtool: AQBRd61lvM2EMxAAJEXKfVtraFTts7X4b2k23g== 2024-01-21T19:58:10.224 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-01-21T19:58:10.502 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-authtool: AQBSd61lRZ66HRAAkznvlNvJ2/Y/j4+BG3keWw== 2024-01-21T19:58:10.832 INFO:teuthology.orchestra.run.smithi053.stderr:Creating initial monmap... 2024-01-21T19:58:10.832 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmpb05irmii:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid 57f6722c-b897-11ee-95b0-87774f69a715 --addv smithi053 [v2:172.21.15.53:3300,v1:172.21.15.53:6789] /tmp/monmap 2024-01-21T19:58:11.094 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-01-21T19:58:11.094 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 57f6722c-b897-11ee-95b0-87774f69a715 2024-01-21T19:58:11.094 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-01-21T19:58:11.452 INFO:teuthology.orchestra.run.smithi053.stderr:monmaptool for smithi053 [v2:172.21.15.53:3300,v1:172.21.15.53:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-01-21T19:58:11.453 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/monmaptool: set fsid to 57f6722c-b897-11ee-95b0-87774f69a715 2024-01-21T19:58:11.453 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-01-21T19:58:11.453 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-01-21T19:58:11.453 INFO:teuthology.orchestra.run.smithi053.stderr:Creating mon... 2024-01-21T19:58:11.453 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/57f6722c-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/57f6722c-b897-11ee-95b0-87774f69a715/mon.smithi053:/var/lib/ceph/mon/ceph-smithi053:z -v /tmp/ceph-tmpv47f4x4q:/tmp/keyring:z -v /tmp/ceph-tmpb05irmii:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi053 --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2024-01-21T19:58:11.804 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.781+0000 7fd6fdeb0700 0 set uid:gid to 167:167 (ceph:ceph) 2024-01-21T19:58:11.804 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.781+0000 7fd6fdeb0700 1 imported monmap: 2024-01-21T19:58:11.804 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: epoch 0 2024-01-21T19:58:11.804 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: fsid 57f6722c-b897-11ee-95b0-87774f69a715 2024-01-21T19:58:11.805 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: last_changed 2024-01-21T19:58:11.102627+0000 2024-01-21T19:58:11.805 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: created 2024-01-21T19:58:11.102627+0000 2024-01-21T19:58:11.805 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2024-01-21T19:58:11.805 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: election_strategy: 1 2024-01-21T19:58:11.805 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.smithi053 2024-01-21T19:58:11.805 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: 2024-01-21T19:58:11.805 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.781+0000 7fd6fdeb0700 0 /usr/bin/ceph-mon: set fsid to 57f6722c-b897-11ee-95b0-87774f69a715 2024-01-21T19:58:11.805 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: RocksDB version: 6.8.1 2024-01-21T19:58:11.805 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: 2024-01-21T19:58:11.805 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-01-21T19:58:11.805 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Compile date May 13 2021 2024-01-21T19:58:11.805 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: DB SUMMARY 2024-01-21T19:58:11.805 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: 2024-01-21T19:58:11.806 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi053/store.db dir, Total Num: 0, files: 2024-01-21T19:58:11.806 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: 2024-01-21T19:58:11.806 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi053/store.db: 2024-01-21T19:58:11.806 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: 2024-01-21T19:58:11.806 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.error_if_exists: 0 2024-01-21T19:58:11.806 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.create_if_missing: 1 2024-01-21T19:58:11.806 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.paranoid_checks: 1 2024-01-21T19:58:11.806 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.env: 0x55ae932281c0 2024-01-21T19:58:11.806 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.fs: Posix File System 2024-01-21T19:58:11.806 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.info_log: 0x55ae94f2eec0 2024-01-21T19:58:11.806 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.max_file_opening_threads: 16 2024-01-21T19:58:11.806 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.statistics: (nil) 2024-01-21T19:58:11.806 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.use_fsync: 0 2024-01-21T19:58:11.806 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.max_log_file_size: 0 2024-01-21T19:58:11.807 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-21T19:58:11.807 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-01-21T19:58:11.807 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.keep_log_file_num: 1000 2024-01-21T19:58:11.807 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.recycle_log_file_num: 0 2024-01-21T19:58:11.807 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.allow_fallocate: 1 2024-01-21T19:58:11.807 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.allow_mmap_reads: 0 2024-01-21T19:58:11.807 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.allow_mmap_writes: 0 2024-01-21T19:58:11.807 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.use_direct_reads: 0 2024-01-21T19:58:11.807 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-21T19:58:11.807 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.create_missing_column_families: 0 2024-01-21T19:58:11.807 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.db_log_dir: 2024-01-21T19:58:11.807 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi053/store.db 2024-01-21T19:58:11.807 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-01-21T19:58:11.807 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.max_subcompactions: 1 2024-01-21T19:58:11.808 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.max_background_flushes: -1 2024-01-21T19:58:11.808 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-01-21T19:58:11.808 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-01-21T19:58:11.808 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-21T19:58:11.808 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-21T19:58:11.808 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-01-21T19:58:11.808 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.advise_random_on_open: 1 2024-01-21T19:58:11.808 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.db_write_buffer_size: 0 2024-01-21T19:58:11.808 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.write_buffer_manager: 0x55ae94f35530 2024-01-21T19:58:11.808 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-21T19:58:11.808 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-01-21T19:58:11.808 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-21T19:58:11.808 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-01-21T19:58:11.809 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.rate_limiter: (nil) 2024-01-21T19:58:11.809 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-21T19:58:11.809 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.wal_recovery_mode: 2 2024-01-21T19:58:11.809 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.enable_thread_tracking: 0 2024-01-21T19:58:11.809 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.enable_pipelined_write: 0 2024-01-21T19:58:11.809 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.unordered_write: 0 2024-01-21T19:58:11.809 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-21T19:58:11.809 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-21T19:58:11.809 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-21T19:58:11.809 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-21T19:58:11.809 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.row_cache: None 2024-01-21T19:58:11.809 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.wal_filter: None 2024-01-21T19:58:11.809 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-21T19:58:11.810 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.allow_ingest_behind: 0 2024-01-21T19:58:11.810 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.preserve_deletes: 0 2024-01-21T19:58:11.810 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.two_write_queues: 0 2024-01-21T19:58:11.810 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.manual_wal_flush: 0 2024-01-21T19:58:11.810 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.atomic_flush: 0 2024-01-21T19:58:11.810 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-21T19:58:11.810 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-01-21T19:58:11.810 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-01-21T19:58:11.810 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.log_readahead_size: 0 2024-01-21T19:58:11.810 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-01-21T19:58:11.810 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.max_background_jobs: 2 2024-01-21T19:58:11.810 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.max_background_compactions: -1 2024-01-21T19:58:11.810 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-21T19:58:11.810 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-21T19:58:11.810 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-01-21T19:58:11.811 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.max_total_wal_size: 0 2024-01-21T19:58:11.811 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-21T19:58:11.811 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-01-21T19:58:11.811 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-01-21T19:58:11.811 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-21T19:58:11.811 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.max_open_files: -1 2024-01-21T19:58:11.811 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.bytes_per_sync: 0 2024-01-21T19:58:11.811 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-01-21T19:58:11.811 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-01-21T19:58:11.811 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Options.compaction_readahead_size: 0 2024-01-21T19:58:11.811 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Compression algorithms supported: 2024-01-21T19:58:11.811 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-21T19:58:11.811 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: kZSTD supported: 0 2024-01-21T19:58:11.811 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: kXpressCompression supported: 0 2024-01-21T19:58:11.811 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: kLZ4HCCompression supported: 1 2024-01-21T19:58:11.812 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: kLZ4Compression supported: 1 2024-01-21T19:58:11.812 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: kBZip2Compression supported: 0 2024-01-21T19:58:11.812 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: kZlibCompression supported: 1 2024-01-21T19:58:11.812 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: kSnappyCompression supported: 1 2024-01-21T19:58:11.812 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-01-21T19:58:11.812 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.782+0000 7fd6fdeb0700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2024-01-21T19:58:11.812 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: 2024-01-21T19:58:11.812 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi053/store.db/MANIFEST-000001 2024-01-21T19:58:11.812 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: 2024-01-21T19:58:11.812 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-01-21T19:58:11.812 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: 2024-01-21T19:58:11.812 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-21T19:58:11.812 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.merge_operator: 2024-01-21T19:58:11.812 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.compaction_filter: None 2024-01-21T19:58:11.813 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.compaction_filter_factory: None 2024-01-21T19:58:11.813 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-01-21T19:58:11.813 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-01-21T19:58:11.813 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ae94e46120) 2024-01-21T19:58:11.813 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2024-01-21T19:58:11.813 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-21T19:58:11.813 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-21T19:58:11.813 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2024-01-21T19:58:11.813 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: index_type: 0 2024-01-21T19:58:11.813 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2024-01-21T19:58:11.813 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: index_shortening: 1 2024-01-21T19:58:11.813 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2024-01-21T19:58:11.813 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2024-01-21T19:58:11.813 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: checksum: 1 2024-01-21T19:58:11.813 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2024-01-21T19:58:11.814 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: block_cache: 0x55ae94e7cd10 2024-01-21T19:58:11.814 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2024-01-21T19:58:11.814 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: block_cache_options: 2024-01-21T19:58:11.814 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: capacity : 536870912 2024-01-21T19:58:11.814 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2024-01-21T19:58:11.814 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2024-01-21T19:58:11.814 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2024-01-21T19:58:11.814 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2024-01-21T19:58:11.814 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2024-01-21T19:58:11.814 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: block_size: 4096 2024-01-21T19:58:11.814 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2024-01-21T19:58:11.814 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2024-01-21T19:58:11.814 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2024-01-21T19:58:11.814 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2024-01-21T19:58:11.814 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: partition_filters: 0 2024-01-21T19:58:11.814 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2024-01-21T19:58:11.815 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2024-01-21T19:58:11.815 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2024-01-21T19:58:11.815 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: verify_compression: 0 2024-01-21T19:58:11.815 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2024-01-21T19:58:11.815 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: format_version: 2 2024-01-21T19:58:11.815 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2024-01-21T19:58:11.815 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: block_align: 0 2024-01-21T19:58:11.815 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: 2024-01-21T19:58:11.815 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.write_buffer_size: 33554432 2024-01-21T19:58:11.815 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.max_write_buffer_number: 2 2024-01-21T19:58:11.815 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.compression: NoCompression 2024-01-21T19:58:11.815 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.bottommost_compression: Disabled 2024-01-21T19:58:11.815 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.prefix_extractor: nullptr 2024-01-21T19:58:11.815 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-21T19:58:11.815 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.num_levels: 7 2024-01-21T19:58:11.816 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-21T19:58:11.816 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-21T19:58:11.816 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-21T19:58:11.816 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-21T19:58:11.816 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-21T19:58:11.816 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-21T19:58:11.816 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-21T19:58:11.816 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-21T19:58:11.816 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-21T19:58:11.816 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-01-21T19:58:11.816 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.compression_opts.level: 32767 2024-01-21T19:58:11.816 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.compression_opts.strategy: 0 2024-01-21T19:58:11.816 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-21T19:58:11.816 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-21T19:58:11.816 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.compression_opts.enabled: false 2024-01-21T19:58:11.817 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-21T19:58:11.817 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-21T19:58:11.817 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-21T19:58:11.817 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.target_file_size_base: 67108864 2024-01-21T19:58:11.817 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-01-21T19:58:11.817 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-21T19:58:11.817 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-21T19:58:11.817 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-21T19:58:11.817 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-21T19:58:11.817 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-21T19:58:11.817 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-21T19:58:11.817 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-21T19:58:11.817 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-21T19:58:11.817 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-21T19:58:11.817 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-21T19:58:11.818 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-21T19:58:11.818 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-21T19:58:11.818 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.arena_block_size: 4194304 2024-01-21T19:58:11.818 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-21T19:58:11.818 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-21T19:58:11.818 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-01-21T19:58:11.818 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.disable_auto_compactions: 0 2024-01-21T19:58:11.818 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-21T19:58:11.818 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-21T19:58:11.818 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-21T19:58:11.818 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-21T19:58:11.818 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-21T19:58:11.818 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-21T19:58:11.818 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-21T19:58:11.818 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-21T19:58:11.819 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-21T19:58:11.819 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-21T19:58:11.819 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.table_properties_collectors: 2024-01-21T19:58:11.819 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.inplace_update_support: 0 2024-01-21T19:58:11.819 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-01-21T19:58:11.819 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-21T19:58:11.819 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-21T19:58:11.819 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-01-21T19:58:11.819 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.bloom_locality: 0 2024-01-21T19:58:11.819 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.max_successive_merges: 0 2024-01-21T19:58:11.819 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-01-21T19:58:11.819 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.paranoid_file_checks: 0 2024-01-21T19:58:11.819 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.force_consistency_checks: 0 2024-01-21T19:58:11.819 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.report_bg_io_stats: 0 2024-01-21T19:58:11.819 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.ttl: 2592000 2024-01-21T19:58:11.820 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-01-21T19:58:11.820 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi053/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-21T19:58:11.820 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: 2024-01-21T19:58:11.820 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.783+0000 7fd6fdeb0700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-01-21T19:58:11.820 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: 2024-01-21T19:58:11.820 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.784+0000 7fd6fdeb0700 4 rocksdb: DB pointer 0x55ae94f41800 2024-01-21T19:58:11.820 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.784+0000 7fd6e6d49700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-01-21T19:58:11.820 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.784+0000 7fd6e6d49700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-01-21T19:58:11.820 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2024-01-21T19:58:11.820 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-21T19:58:11.820 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-21T19:58:11.820 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-21T19:58:11.820 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-21T19:58:11.820 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-21T19:58:11.821 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-01-21T19:58:11.821 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-21T19:58:11.821 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: 2024-01-21T19:58:11.821 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-21T19:58:11.821 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T19:58:11.821 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T19:58:11.821 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-21T19:58:11.821 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-21T19:58:11.821 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: 2024-01-21T19:58:11.821 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-21T19:58:11.821 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T19:58:11.821 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T19:58:11.821 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-21T19:58:11.821 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-01-21T19:58:11.822 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-21T19:58:11.822 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-01-21T19:58:11.822 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-01-21T19:58:11.822 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-01-21T19:58:11.822 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T19:58:11.822 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T19:58:11.822 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-21T19:58:11.822 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: 2024-01-21T19:58:11.822 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-01-21T19:58:11.822 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: 2024-01-21T19:58:11.822 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-21T19:58:11.822 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T19:58:11.822 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T19:58:11.822 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-21T19:58:11.822 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-21T19:58:11.823 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: 2024-01-21T19:58:11.823 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-21T19:58:11.823 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T19:58:11.823 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T19:58:11.823 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-21T19:58:11.823 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-01-21T19:58:11.823 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-21T19:58:11.823 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-01-21T19:58:11.823 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-01-21T19:58:11.823 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-01-21T19:58:11.823 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T19:58:11.823 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T19:58:11.823 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-21T19:58:11.823 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: 2024-01-21T19:58:11.823 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-01-21T19:58:11.824 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: 2024-01-21T19:58:11.824 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.785+0000 7fd6fdeb0700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2024-01-21T19:58:11.824 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.785+0000 7fd6fdeb0700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2024-01-21T19:58:11.824 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:58:11.785+0000 7fd6fdeb0700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi053 for mon.smithi053 2024-01-21T19:58:12.164 INFO:teuthology.orchestra.run.smithi053.stderr:create mon.smithi053 on 2024-01-21T19:58:12.164 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/57f6722c-b897-11ee-95b0-87774f69a715 2024-01-21T19:58:12.165 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl disable ceph.target 2024-01-21T19:58:12.172 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-01-21T19:58:12.259 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl enable ceph.target 2024-01-21T19:58:12.266 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-01-21T19:58:12.353 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl start ceph.target 2024-01-21T19:58:12.360 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl enable ceph-57f6722c-b897-11ee-95b0-87774f69a715.target 2024-01-21T19:58:12.366 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-57f6722c-b897-11ee-95b0-87774f69a715.target → /etc/systemd/system/ceph-57f6722c-b897-11ee-95b0-87774f69a715.target. 2024-01-21T19:58:12.366 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-57f6722c-b897-11ee-95b0-87774f69a715.target → /etc/systemd/system/ceph-57f6722c-b897-11ee-95b0-87774f69a715.target. 2024-01-21T19:58:12.452 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl start ceph-57f6722c-b897-11ee-95b0-87774f69a715.target 2024-01-21T19:58:12.459 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl daemon-reload 2024-01-21T19:58:12.552 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl stop ceph-57f6722c-b897-11ee-95b0-87774f69a715@mon.smithi053 2024-01-21T19:58:12.588 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl reset-failed ceph-57f6722c-b897-11ee-95b0-87774f69a715@mon.smithi053 2024-01-21T19:58:12.593 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: Failed to reset failed state of unit ceph-57f6722c-b897-11ee-95b0-87774f69a715@mon.smithi053.service: Unit ceph-57f6722c-b897-11ee-95b0-87774f69a715@mon.smithi053.service not loaded. 2024-01-21T19:58:12.594 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl enable ceph-57f6722c-b897-11ee-95b0-87774f69a715@mon.smithi053 2024-01-21T19:58:12.601 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: Created symlink /etc/systemd/system/ceph-57f6722c-b897-11ee-95b0-87774f69a715.target.wants/ceph-57f6722c-b897-11ee-95b0-87774f69a715@mon.smithi053.service → /etc/systemd/system/ceph-57f6722c-b897-11ee-95b0-87774f69a715@.service. 2024-01-21T19:58:12.690 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl start ceph-57f6722c-b897-11ee-95b0-87774f69a715@mon.smithi053 2024-01-21T19:58:13.006 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-21T19:58:13.015 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: disabled 2024-01-21T19:58:13.015 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl is-active firewalld.service 2024-01-21T19:58:13.022 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: inactive 2024-01-21T19:58:13.023 INFO:teuthology.orchestra.run.smithi053.stderr:firewalld.service is not enabled 2024-01-21T19:58:13.023 INFO:teuthology.orchestra.run.smithi053.stderr:Not possible to enable service . firewalld.service is not available 2024-01-21T19:58:13.023 INFO:teuthology.orchestra.run.smithi053.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-01-21T19:58:13.024 INFO:teuthology.orchestra.run.smithi053.stderr:Waiting for mon to start... 2024-01-21T19:58:13.024 INFO:teuthology.orchestra.run.smithi053.stderr:Waiting for mon... 2024-01-21T19:58:13.024 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/57f6722c-b897-11ee-95b0-87774f69a715/mon.smithi053:/var/lib/ceph/mon/ceph-smithi053:z -v /tmp/ceph-tmpw7bt2for:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp987f45g0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2024-01-21T19:58:13.423 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.427+0000 7f5a052bb700 1 Processor -- start 2024-01-21T19:58:13.423 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.427+0000 7f5a052bb700 1 -- start start 2024-01-21T19:58:13.424 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.427+0000 7f5a052bb700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a00070460 0x7f5a000fe930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:13.424 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.427+0000 7f5a052bb700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a000fee20 con 0x7f5a00070460 2024-01-21T19:58:13.424 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.427+0000 7f59feffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a00070460 0x7f5a000fe930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:13.424 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.427+0000 7f59feffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a00070460 0x7f5a000fe930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33278/0 (socket says 172.21.15.53:33278) 2024-01-21T19:58:13.424 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.427+0000 7f59feffd700 1 -- 172.21.15.53:0/3292920779 learned_addr learned my addr 172.21.15.53:0/3292920779 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:58:13.424 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.428+0000 7f59feffd700 1 -- 172.21.15.53:0/3292920779 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a000fef60 con 0x7f5a00070460 2024-01-21T19:58:13.425 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.428+0000 7f59feffd700 1 --2- 172.21.15.53:0/3292920779 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a00070460 0x7f5a000fe930 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f59f000b5f0 tx=0x7f59f0006ee0).ready entity=mon.0 client_cookie=6f21b651600c224c server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:13.425 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.428+0000 7f59fdffb700 1 -- 172.21.15.53:0/3292920779 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59f000d600 con 0x7f5a00070460 2024-01-21T19:58:13.425 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.428+0000 7f59fdffb700 1 -- 172.21.15.53:0/3292920779 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59f000d760 con 0x7f5a00070460 2024-01-21T19:58:13.425 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.428+0000 7f59fdffb700 1 -- 172.21.15.53:0/3292920779 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59f000da30 con 0x7f5a00070460 2024-01-21T19:58:13.425 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.428+0000 7f5a052bb700 1 -- 172.21.15.53:0/3292920779 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a00070460 msgr2=0x7f5a000fe930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:13.425 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.428+0000 7f5a052bb700 1 --2- 172.21.15.53:0/3292920779 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a00070460 0x7f5a000fe930 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f59f000b5f0 tx=0x7f59f0006ee0).stop 2024-01-21T19:58:13.425 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.429+0000 7f5a052bb700 1 -- 172.21.15.53:0/3292920779 shutdown_connections 2024-01-21T19:58:13.425 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.429+0000 7f5a052bb700 1 --2- 172.21.15.53:0/3292920779 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a00070460 0x7f5a000fe930 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:13.425 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.429+0000 7f5a052bb700 1 -- 172.21.15.53:0/3292920779 >> 172.21.15.53:0/3292920779 conn(0x7f5a000f63b0 msgr2=0x7f5a000f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:58:13.425 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.429+0000 7f5a052bb700 1 -- 172.21.15.53:0/3292920779 shutdown_connections 2024-01-21T19:58:13.426 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.429+0000 7f5a052bb700 1 -- 172.21.15.53:0/3292920779 wait complete. 2024-01-21T19:58:13.426 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.429+0000 7f5a052bb700 1 Processor -- start 2024-01-21T19:58:13.426 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.429+0000 7f5a052bb700 1 -- start start 2024-01-21T19:58:13.426 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.430+0000 7f5a052bb700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a00070460 0x7f5a0010a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:13.426 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.430+0000 7f5a052bb700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0010a780 con 0x7f5a00070460 2024-01-21T19:58:13.426 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.430+0000 7f59feffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a00070460 0x7f5a0010a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:13.426 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.430+0000 7f59feffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a00070460 0x7f5a0010a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33282/0 (socket says 172.21.15.53:33282) 2024-01-21T19:58:13.426 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.430+0000 7f59feffd700 1 -- 172.21.15.53:0/1289650461 learned_addr learned my addr 172.21.15.53:0/1289650461 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:58:13.426 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.430+0000 7f59feffd700 1 -- 172.21.15.53:0/1289650461 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59f000b200 con 0x7f5a00070460 2024-01-21T19:58:13.426 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.430+0000 7f59feffd700 1 --2- 172.21.15.53:0/1289650461 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a00070460 0x7f5a0010a290 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f59f0006c70 tx=0x7f59f0016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:13.426 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.430+0000 7f59f77fe700 1 -- 172.21.15.53:0/1289650461 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59f0011b40 con 0x7f5a00070460 2024-01-21T19:58:13.426 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.430+0000 7f59f77fe700 1 -- 172.21.15.53:0/1289650461 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59f0011ca0 con 0x7f5a00070460 2024-01-21T19:58:13.427 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.430+0000 7f5a052bb700 1 -- 172.21.15.53:0/1289650461 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a0010a980 con 0x7f5a00070460 2024-01-21T19:58:13.427 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.430+0000 7f59f77fe700 1 -- 172.21.15.53:0/1289650461 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59f001d430 con 0x7f5a00070460 2024-01-21T19:58:13.427 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.430+0000 7f5a052bb700 1 -- 172.21.15.53:0/1289650461 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a0010ad90 con 0x7f5a00070460 2024-01-21T19:58:13.427 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.431+0000 7f59f77fe700 1 -- 172.21.15.53:0/1289650461 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f59f000f070 con 0x7f5a00070460 2024-01-21T19:58:13.427 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.431+0000 7f59f77fe700 1 -- 172.21.15.53:0/1289650461 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f59f0027460 con 0x7f5a00070460 2024-01-21T19:58:13.427 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.432+0000 7f5a052bb700 1 -- 172.21.15.53:0/1289650461 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a00045ad0 con 0x7f5a00070460 2024-01-21T19:58:13.427 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.434+0000 7f59f77fe700 1 -- 172.21.15.53:0/1289650461 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f59f0016070 con 0x7f5a00070460 2024-01-21T19:58:13.434 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:13 smithi053 bash[38417]: 35fac4baaec77f497530471bc36bb6275dad492488b5f83dab906fca99254850 2024-01-21T19:58:13.434 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:13 smithi053 systemd[1]: Started Ceph mon.smithi053 for 57f6722c-b897-11ee-95b0-87774f69a715. 2024-01-21T19:58:13.499 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.504+0000 7f5a052bb700 1 -- 172.21.15.53:0/1289650461 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f5a00103e70 con 0x7f5a00070460 2024-01-21T19:58:13.499 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.505+0000 7f59f77fe700 1 -- 172.21.15.53:0/1289650461 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f59f0036070 con 0x7f5a00070460 2024-01-21T19:58:13.499 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: cluster: 2024-01-21T19:58:13.499 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: id: 57f6722c-b897-11ee-95b0-87774f69a715 2024-01-21T19:58:13.500 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-01-21T19:58:13.500 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.500 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: services: 2024-01-21T19:58:13.500 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi053 (age 0.501259s) 2024-01-21T19:58:13.500 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: mgr: no daemons active 2024-01-21T19:58:13.500 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-01-21T19:58:13.500 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.500 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: data: 2024-01-21T19:58:13.500 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-01-21T19:58:13.500 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-01-21T19:58:13.500 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-01-21T19:58:13.500 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: pgs: 2024-01-21T19:58:13.500 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.500 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.505+0000 7f5a052bb700 1 -- 172.21.15.53:0/1289650461 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a00070460 msgr2=0x7f5a0010a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:13.501 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.505+0000 7f5a052bb700 1 --2- 172.21.15.53:0/1289650461 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a00070460 0x7f5a0010a290 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f59f0006c70 tx=0x7f59f0016000).stop 2024-01-21T19:58:13.501 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.505+0000 7f5a052bb700 1 -- 172.21.15.53:0/1289650461 shutdown_connections 2024-01-21T19:58:13.501 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.505+0000 7f5a052bb700 1 --2- 172.21.15.53:0/1289650461 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a00070460 0x7f5a0010a290 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:13.501 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.505+0000 7f5a052bb700 1 -- 172.21.15.53:0/1289650461 >> 172.21.15.53:0/1289650461 conn(0x7f5a000f63b0 msgr2=0x7f5a000f80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:58:13.501 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.505+0000 7f5a052bb700 1 -- 172.21.15.53:0/1289650461 shutdown_connections 2024-01-21T19:58:13.501 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:13.505+0000 7f5a052bb700 1 -- 172.21.15.53:0/1289650461 wait complete. 2024-01-21T19:58:13.854 INFO:teuthology.orchestra.run.smithi053.stderr:mon is available 2024-01-21T19:58:13.854 INFO:teuthology.orchestra.run.smithi053.stderr:Assimilating anything we can from ceph.conf... 2024-01-21T19:58:13.854 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/57f6722c-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpw7bt2for:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp987f45g0:/etc/ceph/ceph.conf:z -v /var/lib/ceph/57f6722c-b897-11ee-95b0-87774f69a715/mon.smithi053:/var/lib/ceph/mon/ceph-smithi053:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi053/config 2024-01-21T19:58:14.258 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.262+0000 7fe2a5343700 1 Processor -- start 2024-01-21T19:58:14.259 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.262+0000 7fe2a5343700 1 -- start start 2024-01-21T19:58:14.259 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.262+0000 7fe2a5343700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2a00fab80 0x7fe2a00fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:14.259 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.262+0000 7fe2a5343700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a00fb490 con 0x7fe2a00fab80 2024-01-21T19:58:14.259 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.263+0000 7fe29effd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2a00fab80 0x7fe2a00fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:14.259 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.263+0000 7fe29effd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2a00fab80 0x7fe2a00fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33292/0 (socket says 172.21.15.53:33292) 2024-01-21T19:58:14.259 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.263+0000 7fe29effd700 1 -- 172.21.15.53:0/3357892448 learned_addr learned my addr 172.21.15.53:0/3357892448 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:58:14.259 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.263+0000 7fe29effd700 1 -- 172.21.15.53:0/3357892448 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2a00fb5d0 con 0x7fe2a00fab80 2024-01-21T19:58:14.259 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.263+0000 7fe29effd700 1 --2- 172.21.15.53:0/3357892448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2a00fab80 0x7fe2a00fafa0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fe29400b870 tx=0x7fe294006f70).ready entity=mon.0 client_cookie=5b378e4fcbe68b2 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:14.259 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.264+0000 7fe29dffb700 1 -- 172.21.15.53:0/3357892448 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe29400da20 con 0x7fe2a00fab80 2024-01-21T19:58:14.259 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.264+0000 7fe29dffb700 1 -- 172.21.15.53:0/3357892448 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe29400db80 con 0x7fe2a00fab80 2024-01-21T19:58:14.259 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.264+0000 7fe29dffb700 1 -- 172.21.15.53:0/3357892448 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe29400de50 con 0x7fe2a00fab80 2024-01-21T19:58:14.259 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.264+0000 7fe2a5343700 1 -- 172.21.15.53:0/3357892448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2a00fab80 msgr2=0x7fe2a00fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:14.260 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.264+0000 7fe2a5343700 1 --2- 172.21.15.53:0/3357892448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2a00fab80 0x7fe2a00fafa0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fe29400b870 tx=0x7fe294006f70).stop 2024-01-21T19:58:14.260 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.264+0000 7fe2a5343700 1 -- 172.21.15.53:0/3357892448 shutdown_connections 2024-01-21T19:58:14.260 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.264+0000 7fe2a5343700 1 --2- 172.21.15.53:0/3357892448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2a00fab80 0x7fe2a00fafa0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:14.260 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.264+0000 7fe2a5343700 1 -- 172.21.15.53:0/3357892448 >> 172.21.15.53:0/3357892448 conn(0x7fe2a00f63c0 msgr2=0x7fe2a00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:58:14.260 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.264+0000 7fe2a5343700 1 -- 172.21.15.53:0/3357892448 shutdown_connections 2024-01-21T19:58:14.260 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.264+0000 7fe2a5343700 1 -- 172.21.15.53:0/3357892448 wait complete. 2024-01-21T19:58:14.260 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.265+0000 7fe2a5343700 1 Processor -- start 2024-01-21T19:58:14.260 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.265+0000 7fe2a5343700 1 -- start start 2024-01-21T19:58:14.260 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.265+0000 7fe2a5343700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2a00fab80 0x7fe2a010c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:14.260 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.265+0000 7fe2a5343700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a010c940 con 0x7fe2a00fab80 2024-01-21T19:58:14.260 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.265+0000 7fe29effd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2a00fab80 0x7fe2a010c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:14.260 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.265+0000 7fe29effd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2a00fab80 0x7fe2a010c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33304/0 (socket says 172.21.15.53:33304) 2024-01-21T19:58:14.260 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.265+0000 7fe29effd700 1 -- 172.21.15.53:0/3866529046 learned_addr learned my addr 172.21.15.53:0/3866529046 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:58:14.261 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.265+0000 7fe29effd700 1 -- 172.21.15.53:0/3866529046 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe29400b480 con 0x7fe2a00fab80 2024-01-21T19:58:14.261 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.266+0000 7fe29effd700 1 --2- 172.21.15.53:0/3866529046 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2a00fab80 0x7fe2a010c450 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fe294006c70 tx=0x7fe294006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:14.261 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.266+0000 7fe2877fe700 1 -- 172.21.15.53:0/3866529046 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe294019450 con 0x7fe2a00fab80 2024-01-21T19:58:14.261 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.266+0000 7fe2877fe700 1 -- 172.21.15.53:0/3866529046 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2940195b0 con 0x7fe2a00fab80 2024-01-21T19:58:14.261 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.266+0000 7fe2a5343700 1 -- 172.21.15.53:0/3866529046 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2a010cb40 con 0x7fe2a00fab80 2024-01-21T19:58:14.261 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.266+0000 7fe2877fe700 1 -- 172.21.15.53:0/3866529046 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe294019880 con 0x7fe2a00fab80 2024-01-21T19:58:14.261 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.266+0000 7fe2a5343700 1 -- 172.21.15.53:0/3866529046 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2a010cf50 con 0x7fe2a00fab80 2024-01-21T19:58:14.261 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.266+0000 7fe2877fe700 1 -- 172.21.15.53:0/3866529046 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fe29402a070 con 0x7fe2a00fab80 2024-01-21T19:58:14.261 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.266+0000 7fe2877fe700 1 -- 172.21.15.53:0/3866529046 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fe2940239b0 con 0x7fe2a00fab80 2024-01-21T19:58:14.262 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.267+0000 7fe2a5343700 1 -- 172.21.15.53:0/3866529046 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2a0045ad0 con 0x7fe2a00fab80 2024-01-21T19:58:14.262 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.269+0000 7fe2877fe700 1 -- 172.21.15.53:0/3866529046 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fe2940429c0 con 0x7fe2a00fab80 2024-01-21T19:58:14.328 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.336+0000 7fe2a5343700 1 -- 172.21.15.53:0/3866529046 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fe2a00fbec0 con 0x7fe2a00fab80 2024-01-21T19:58:14.335 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.339+0000 7fe2877fe700 1 -- 172.21.15.53:0/3866529046 <== mon.0 v2:172.21.15.53:3300/0 7 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe29400f840 con 0x7fe2a00fab80 2024-01-21T19:58:14.335 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.339+0000 7fe2877fe700 1 -- 172.21.15.53:0/3866529046 <== mon.0 v2:172.21.15.53:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+464 (secure 0 0 0) 0x7fe294031030 con 0x7fe2a00fab80 2024-01-21T19:58:14.335 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.335 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: [global] 2024-01-21T19:58:14.335 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: fsid = 57f6722c-b897-11ee-95b0-87774f69a715 2024-01-21T19:58:14.335 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.53:3300,v1:172.21.15.53:6789] 2024-01-21T19:58:14.335 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-01-21T19:58:14.336 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-01-21T19:58:14.336 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-01-21T19:58:14.336 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-01-21T19:58:14.336 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.336 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: [mgr] 2024-01-21T19:58:14.336 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-01-21T19:58:14.336 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.336 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: [osd] 2024-01-21T19:58:14.336 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2024-01-21T19:58:14.336 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2024-01-21T19:58:14.337 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2024-01-21T19:58:14.337 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-01-21T19:58:14.337 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2024-01-21T19:58:14.337 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-01-21T19:58:14.337 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.340+0000 7fe2a5343700 1 -- 172.21.15.53:0/3866529046 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2a00fab80 msgr2=0x7fe2a010c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:14.337 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.340+0000 7fe2a5343700 1 --2- 172.21.15.53:0/3866529046 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2a00fab80 0x7fe2a010c450 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fe294006c70 tx=0x7fe294006f70).stop 2024-01-21T19:58:14.337 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.340+0000 7fe2a5343700 1 -- 172.21.15.53:0/3866529046 shutdown_connections 2024-01-21T19:58:14.337 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.340+0000 7fe2a5343700 1 --2- 172.21.15.53:0/3866529046 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2a00fab80 0x7fe2a010c450 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:14.337 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.340+0000 7fe2a5343700 1 -- 172.21.15.53:0/3866529046 >> 172.21.15.53:0/3866529046 conn(0x7fe2a00f63c0 msgr2=0x7fe2a00f85c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:58:14.337 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.340+0000 7fe2a5343700 1 -- 172.21.15.53:0/3866529046 shutdown_connections 2024-01-21T19:58:14.338 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:14.340+0000 7fe2a5343700 1 -- 172.21.15.53:0/3866529046 wait complete. 2024-01-21T19:58:14.351 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:14 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38432]: cluster 2024-01-21T19:58:13.004893+0000 mon.smithi053 (mon.0) 2 : cluster [INF] mon.smithi053 is new leader, mons smithi053 in quorum (ranks 0) 2024-01-21T19:58:14.351 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:14 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38432]: cluster 2024-01-21T19:58:13.005023+0000 mon.smithi053 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi053=[v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0]} 2024-01-21T19:58:14.351 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:14 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38432]: cluster 2024-01-21T19:58:13.006701+0000 mon.smithi053 (mon.0) 4 : cluster [DBG] fsmap 2024-01-21T19:58:14.351 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:14 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38432]: cluster 2024-01-21T19:58:13.008576+0000 mon.smithi053 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-01-21T19:58:14.351 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:14 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38432]: cluster 2024-01-21T19:58:13.009030+0000 mon.smithi053 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2024-01-21T19:58:14.351 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:14 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38432]: audit 2024-01-21T19:58:13.506119+0000 mon.smithi053 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.53:0/1289650461' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-01-21T19:58:14.704 INFO:teuthology.orchestra.run.smithi053.stderr:Generating new minimal ceph.conf... 2024-01-21T19:58:14.704 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/57f6722c-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpw7bt2for:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp987f45g0:/etc/ceph/ceph.conf:z -v /var/lib/ceph/57f6722c-b897-11ee-95b0-87774f69a715/mon.smithi053:/var/lib/ceph/mon/ceph-smithi053:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi053/config 2024-01-21T19:58:15.059 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:15.066+0000 7fedcf59e700 1 Processor -- start 2024-01-21T19:58:15.059 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:15.066+0000 7fedcf59e700 1 -- start start 2024-01-21T19:58:15.059 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:15.066+0000 7fedcf59e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fedd00703b0 0x7fedd0100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:15.059 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:15.066+0000 7fedcf59e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd00fc5a0 con 0x7fedd00703b0 2024-01-21T19:58:15.059 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:15.066+0000 7fedce59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fedd00703b0 0x7fedd0100bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:15.059 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:15.066+0000 7fedce59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fedd00703b0 0x7fedd0100bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33308/0 (socket says 172.21.15.53:33308) 2024-01-21T19:58:15.059 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:15.066+0000 7fedce59c700 1 -- 172.21.15.53:0/3150945907 learned_addr learned my addr 172.21.15.53:0/3150945907 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:58:15.060 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:15.066+0000 7fedce59c700 1 -- 172.21.15.53:0/3150945907 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd01010a0 con 0x7fedd00703b0 2024-01-21T19:58:15.060 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:15.067+0000 7fedce59c700 1 --2- 172.21.15.53:0/3150945907 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fedd00703b0 0x7fedd0100bb0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fedc000b5f0 tx=0x7fedc0006ee0).ready entity=mon.0 client_cookie=5eceb34fad832cf9 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:15.061 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:15.067+0000 7fedcd59a700 1 -- 172.21.15.53:0/3150945907 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fedc000d600 con 0x7fedd00703b0 2024-01-21T19:58:15.061 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:15.067+0000 7fedcd59a700 1 -- 172.21.15.53:0/3150945907 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fedc000dbd0 con 0x7fedd00703b0 2024-01-21T19:58:15.062 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:15.067+0000 7fedcd59a700 1 -- 172.21.15.53:0/3150945907 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fedc0011c00 con 0x7fedd00703b0 2024-01-21T19:58:15.062 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:15.067+0000 7fedcf59e700 1 -- 172.21.15.53:0/3150945907 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fedd00703b0 msgr2=0x7fedd0100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:15.062 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:15.067+0000 7fedcf59e700 1 --2- 172.21.15.53:0/3150945907 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fedd00703b0 0x7fedd0100bb0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fedc000b5f0 tx=0x7fedc0006ee0).stop 2024-01-21T19:58:15.062 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:15.067+0000 7fedcf59e700 1 -- 172.21.15.53:0/3150945907 shutdown_connections 2024-01-21T19:58:15.062 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:15.067+0000 7fedcf59e700 1 --2- 172.21.15.53:0/3150945907 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fedd00703b0 0x7fedd0100bb0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:15.062 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:15.067+0000 7fedcf59e700 1 -- 172.21.15.53:0/3150945907 >> 172.21.15.53:0/3150945907 conn(0x7fedd00f6210 msgr2=0x7fedd00f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:58:15.062 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:15.067+0000 7fedcf59e700 1 -- 172.21.15.53:0/3150945907 shutdown_connections 2024-01-21T19:58:15.062 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:15.067+0000 7fedcf59e700 1 -- 172.21.15.53:0/3150945907 wait complete. 2024-01-21T19:58:15.062 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:15.068+0000 7fedcf59e700 1 Processor -- start 2024-01-21T19:58:15.062 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:15.068+0000 7fedcf59e700 1 -- start start 2024-01-21T19:58:15.062 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:15.068+0000 7fedcf59e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fedd00703b0 0x7fedd010ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:15.062 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:15.068+0000 7fedcf59e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd010cf60 con 0x7fedd00703b0 2024-01-21T19:58:15.062 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:15.068+0000 7fedce59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fedd00703b0 0x7fedd010ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:15.062 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:15.068+0000 7fedce59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fedd00703b0 0x7fedd010ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33314/0 (socket says 172.21.15.53:33314) 2024-01-21T19:58:15.063 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:15.068+0000 7fedce59c700 1 -- 172.21.15.53:0/1501910749 learned_addr learned my addr 172.21.15.53:0/1501910749 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:58:15.063 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:15.069+0000 7fedce59c700 1 -- 172.21.15.53:0/1501910749 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedc000b200 con 0x7fedd00703b0 2024-01-21T19:58:15.063 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:15.069+0000 7fedce59c700 1 --2- 172.21.15.53:0/1501910749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fedd00703b0 0x7fedd010ca70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fedc000e040 tx=0x7fedc0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:15.063 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:15.069+0000 7fedbeffd700 1 -- 172.21.15.53:0/1501910749 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fedc000d9f0 con 0x7fedd00703b0 2024-01-21T19:58:15.063 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:15.069+0000 7fedbeffd700 1 -- 172.21.15.53:0/1501910749 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fedc001a9f0 con 0x7fedd00703b0 2024-01-21T19:58:15.063 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:15.069+0000 7fedbeffd700 1 -- 172.21.15.53:0/1501910749 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fedc001b8c0 con 0x7fedd00703b0 2024-01-21T19:58:15.063 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:15.069+0000 7fedcf59e700 1 -- 172.21.15.53:0/1501910749 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedd010d160 con 0x7fedd00703b0 2024-01-21T19:58:15.063 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:15.069+0000 7fedcf59e700 1 -- 172.21.15.53:0/1501910749 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedd010d570 con 0x7fedd00703b0 2024-01-21T19:58:15.063 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:15.069+0000 7fedbeffd700 1 -- 172.21.15.53:0/1501910749 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fedc000f070 con 0x7fedd00703b0 2024-01-21T19:58:15.063 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:15.069+0000 7fedbeffd700 1 -- 172.21.15.53:0/1501910749 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fedc0025ae0 con 0x7fedd00703b0 2024-01-21T19:58:15.064 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:15.071+0000 7fedcf59e700 1 -- 172.21.15.53:0/1501910749 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedd0045ad0 con 0x7fedd00703b0 2024-01-21T19:58:15.065 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:15.073+0000 7fedbeffd700 1 -- 172.21.15.53:0/1501910749 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fedc0016070 con 0x7fedd00703b0 2024-01-21T19:58:15.132 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:15.140+0000 7fedcf59e700 1 -- 172.21.15.53:0/1501910749 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fedd0101c00 con 0x7fedd00703b0 2024-01-21T19:58:15.132 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:15.141+0000 7fedbeffd700 1 -- 172.21.15.53:0/1501910749 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7fedc001bd70 con 0x7fedd00703b0 2024-01-21T19:58:15.134 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:15.141+0000 7fedcf59e700 1 -- 172.21.15.53:0/1501910749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fedd00703b0 msgr2=0x7fedd010ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:15.134 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:15.141+0000 7fedcf59e700 1 --2- 172.21.15.53:0/1501910749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fedd00703b0 0x7fedd010ca70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fedc000e040 tx=0x7fedc0006ee0).stop 2024-01-21T19:58:15.134 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:15.141+0000 7fedcf59e700 1 -- 172.21.15.53:0/1501910749 shutdown_connections 2024-01-21T19:58:15.134 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:15.141+0000 7fedcf59e700 1 --2- 172.21.15.53:0/1501910749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fedd00703b0 0x7fedd010ca70 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:15.134 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:15.141+0000 7fedcf59e700 1 -- 172.21.15.53:0/1501910749 >> 172.21.15.53:0/1501910749 conn(0x7fedd00f6210 msgr2=0x7fedd00f7d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:58:15.134 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:15.141+0000 7fedcf59e700 1 -- 172.21.15.53:0/1501910749 shutdown_connections 2024-01-21T19:58:15.134 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:15.141+0000 7fedcf59e700 1 -- 172.21.15.53:0/1501910749 wait complete. 2024-01-21T19:58:15.336 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:15 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38432]: audit 2024-01-21T19:58:14.337612+0000 mon.smithi053 (mon.0) 8 : audit [INF] from='client.? 172.21.15.53:0/3866529046' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-21T19:58:15.336 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:15 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38432]: audit 2024-01-21T19:58:14.340675+0000 mon.smithi053 (mon.0) 9 : audit [INF] from='client.? 172.21.15.53:0/3866529046' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-01-21T19:58:15.521 INFO:teuthology.orchestra.run.smithi053.stderr:Restarting the monitor... 2024-01-21T19:58:15.521 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl restart ceph-57f6722c-b897-11ee-95b0-87774f69a715@mon.smithi053 2024-01-21T19:58:15.594 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:15 smithi053 systemd[1]: Stopping Ceph mon.smithi053 for 57f6722c-b897-11ee-95b0-87774f69a715... 2024-01-21T19:58:15.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:15 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38432]: debug 2024-01-21T19:58:15.601+0000 7fc04e903700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi053 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-01-21T19:58:15.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:15 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38432]: debug 2024-01-21T19:58:15.601+0000 7fc04e903700 -1 mon.smithi053@0(leader) e1 *** Got Signal Terminated *** 2024-01-21T19:58:16.306 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 podman[38757]: ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053 2024-01-21T19:58:16.306 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 systemd[1]: ceph-57f6722c-b897-11ee-95b0-87774f69a715@mon.smithi053.service: Succeeded. 2024-01-21T19:58:16.306 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 systemd[1]: Stopped Ceph mon.smithi053 for 57f6722c-b897-11ee-95b0-87774f69a715. 2024-01-21T19:58:16.306 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 systemd[1]: Starting Ceph mon.smithi053 for 57f6722c-b897-11ee-95b0-87774f69a715... 2024-01-21T19:58:16.396 INFO:teuthology.orchestra.run.smithi053.stderr:Setting mon public_network to 172.21.0.0/20 2024-01-21T19:58:16.397 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/57f6722c-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpw7bt2for:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp987f45g0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2024-01-21T19:58:16.685 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 bash[38828]: 81142df0ae0014857ac4255c6f0150d71f541087763f2dee74a543cf182044e2 2024-01-21T19:58:16.686 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.401+0000 7fb6899c9700 0 set uid:gid to 167:167 (ceph:ceph) 2024-01-21T19:58:16.686 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.401+0000 7fb6899c9700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2024-01-21T19:58:16.686 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.401+0000 7fb6899c9700 0 pidfile_write: ignore empty --pid-file 2024-01-21T19:58:16.686 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 systemd[1]: Started Ceph mon.smithi053 for 57f6722c-b897-11ee-95b0-87774f69a715. 2024-01-21T19:58:16.686 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.405+0000 7fb6899c9700 0 load: jerasure load: lrc load: isa 2024-01-21T19:58:16.686 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: RocksDB version: 6.8.1 2024-01-21T19:58:16.686 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: 2024-01-21T19:58:16.686 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-01-21T19:58:16.686 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Compile date May 13 2021 2024-01-21T19:58:16.686 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: DB SUMMARY 2024-01-21T19:58:16.686 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: 2024-01-21T19:58:16.686 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: CURRENT file: CURRENT 2024-01-21T19:58:16.687 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: 2024-01-21T19:58:16.687 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: IDENTITY file: IDENTITY 2024-01-21T19:58:16.687 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: 2024-01-21T19:58:16.687 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2024-01-21T19:58:16.687 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: 2024-01-21T19:58:16.687 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi053/store.db dir, Total Num: 1, files: 000004.sst 2024-01-21T19:58:16.687 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: 2024-01-21T19:58:16.687 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi053/store.db: 000006.log size: 92918 ; 2024-01-21T19:58:16.687 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: 2024-01-21T19:58:16.687 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.error_if_exists: 0 2024-01-21T19:58:16.687 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.create_if_missing: 0 2024-01-21T19:58:16.687 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.paranoid_checks: 1 2024-01-21T19:58:16.687 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.env: 0x563d9a78f1c0 2024-01-21T19:58:16.687 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.fs: Posix File System 2024-01-21T19:58:16.688 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.info_log: 0x563d9bc77960 2024-01-21T19:58:16.688 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.max_file_opening_threads: 16 2024-01-21T19:58:16.688 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.statistics: (nil) 2024-01-21T19:58:16.688 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.use_fsync: 0 2024-01-21T19:58:16.688 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.max_log_file_size: 0 2024-01-21T19:58:16.688 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-21T19:58:16.688 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-01-21T19:58:16.688 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.keep_log_file_num: 1000 2024-01-21T19:58:16.688 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.recycle_log_file_num: 0 2024-01-21T19:58:16.688 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.allow_fallocate: 1 2024-01-21T19:58:16.689 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.allow_mmap_reads: 0 2024-01-21T19:58:16.689 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.allow_mmap_writes: 0 2024-01-21T19:58:16.689 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.use_direct_reads: 0 2024-01-21T19:58:16.689 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-21T19:58:16.689 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.create_missing_column_families: 0 2024-01-21T19:58:16.689 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.db_log_dir: 2024-01-21T19:58:16.689 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi053/store.db 2024-01-21T19:58:16.689 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-01-21T19:58:16.689 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.max_subcompactions: 1 2024-01-21T19:58:16.689 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.max_background_flushes: -1 2024-01-21T19:58:16.689 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-01-21T19:58:16.689 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-01-21T19:58:16.689 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-21T19:58:16.690 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-21T19:58:16.690 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-01-21T19:58:16.690 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.advise_random_on_open: 1 2024-01-21T19:58:16.690 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.db_write_buffer_size: 0 2024-01-21T19:58:16.690 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.write_buffer_manager: 0x563d9c8ec030 2024-01-21T19:58:16.690 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-21T19:58:16.690 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-01-21T19:58:16.690 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-21T19:58:16.690 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-01-21T19:58:16.690 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.rate_limiter: (nil) 2024-01-21T19:58:16.690 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-21T19:58:16.690 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.wal_recovery_mode: 2 2024-01-21T19:58:16.690 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.enable_thread_tracking: 0 2024-01-21T19:58:16.690 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.enable_pipelined_write: 0 2024-01-21T19:58:16.690 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.unordered_write: 0 2024-01-21T19:58:16.690 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-21T19:58:16.691 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-21T19:58:16.691 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-21T19:58:16.691 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-21T19:58:16.691 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.row_cache: None 2024-01-21T19:58:16.691 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.wal_filter: None 2024-01-21T19:58:16.691 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-21T19:58:16.691 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.allow_ingest_behind: 0 2024-01-21T19:58:16.691 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.preserve_deletes: 0 2024-01-21T19:58:16.691 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.two_write_queues: 0 2024-01-21T19:58:16.691 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.manual_wal_flush: 0 2024-01-21T19:58:16.691 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.atomic_flush: 0 2024-01-21T19:58:16.691 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-21T19:58:16.691 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-01-21T19:58:16.691 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-01-21T19:58:16.691 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.log_readahead_size: 0 2024-01-21T19:58:16.691 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-01-21T19:58:16.692 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.max_background_jobs: 2 2024-01-21T19:58:16.692 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.max_background_compactions: -1 2024-01-21T19:58:16.692 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-21T19:58:16.692 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-21T19:58:16.692 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-01-21T19:58:16.692 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.max_total_wal_size: 0 2024-01-21T19:58:16.692 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-21T19:58:16.692 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-01-21T19:58:16.692 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-01-21T19:58:16.692 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-21T19:58:16.692 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.max_open_files: -1 2024-01-21T19:58:16.692 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.bytes_per_sync: 0 2024-01-21T19:58:16.692 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-01-21T19:58:16.692 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-01-21T19:58:16.693 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.compaction_readahead_size: 0 2024-01-21T19:58:16.693 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Compression algorithms supported: 2024-01-21T19:58:16.693 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-21T19:58:16.693 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: kZSTD supported: 0 2024-01-21T19:58:16.693 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: kXpressCompression supported: 0 2024-01-21T19:58:16.693 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: kLZ4HCCompression supported: 1 2024-01-21T19:58:16.693 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: kLZ4Compression supported: 1 2024-01-21T19:58:16.693 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: kBZip2Compression supported: 0 2024-01-21T19:58:16.693 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: kZlibCompression supported: 1 2024-01-21T19:58:16.693 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: kSnappyCompression supported: 1 2024-01-21T19:58:16.693 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-01-21T19:58:16.693 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi053/store.db/MANIFEST-000005 2024-01-21T19:58:16.693 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: 2024-01-21T19:58:16.693 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-01-21T19:58:16.694 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: 2024-01-21T19:58:16.694 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-21T19:58:16.694 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.merge_operator: 2024-01-21T19:58:16.694 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.compaction_filter: None 2024-01-21T19:58:16.694 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.compaction_filter_factory: None 2024-01-21T19:58:16.694 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-01-21T19:58:16.694 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-01-21T19:58:16.694 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x563d9bb8e128) 2024-01-21T19:58:16.694 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cache_index_and_filter_blocks: 1 2024-01-21T19:58:16.694 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-21T19:58:16.694 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-21T19:58:16.694 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: pin_top_level_index_and_filter: 1 2024-01-21T19:58:16.694 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: index_type: 0 2024-01-21T19:58:16.694 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: data_block_index_type: 0 2024-01-21T19:58:16.694 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: index_shortening: 1 2024-01-21T19:58:16.694 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: data_block_hash_table_util_ratio: 0.750000 2024-01-21T19:58:16.695 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: hash_index_allow_collision: 1 2024-01-21T19:58:16.695 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: checksum: 1 2024-01-21T19:58:16.695 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: no_block_cache: 0 2024-01-21T19:58:16.695 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: block_cache: 0x563d9bbc4f10 2024-01-21T19:58:16.695 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: block_cache_name: BinnedLRUCache 2024-01-21T19:58:16.695 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: block_cache_options: 2024-01-21T19:58:16.695 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: capacity : 536870912 2024-01-21T19:58:16.695 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: num_shard_bits : 4 2024-01-21T19:58:16.695 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: strict_capacity_limit : 0 2024-01-21T19:58:16.695 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: high_pri_pool_ratio: 0.000 2024-01-21T19:58:16.695 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: block_cache_compressed: (nil) 2024-01-21T19:58:16.695 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: persistent_cache: (nil) 2024-01-21T19:58:16.695 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: block_size: 4096 2024-01-21T19:58:16.695 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: block_size_deviation: 10 2024-01-21T19:58:16.696 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: block_restart_interval: 16 2024-01-21T19:58:16.696 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: index_block_restart_interval: 1 2024-01-21T19:58:16.696 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: metadata_block_size: 4096 2024-01-21T19:58:16.696 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: partition_filters: 0 2024-01-21T19:58:16.696 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: use_delta_encoding: 1 2024-01-21T19:58:16.696 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: filter_policy: rocksdb.BuiltinBloomFilter 2024-01-21T19:58:16.696 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: whole_key_filtering: 1 2024-01-21T19:58:16.696 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: verify_compression: 0 2024-01-21T19:58:16.696 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: read_amp_bytes_per_bit: 0 2024-01-21T19:58:16.696 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: format_version: 2 2024-01-21T19:58:16.696 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: enable_index_compression: 1 2024-01-21T19:58:16.696 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: block_align: 0 2024-01-21T19:58:16.696 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: 2024-01-21T19:58:16.696 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.write_buffer_size: 33554432 2024-01-21T19:58:16.697 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.max_write_buffer_number: 2 2024-01-21T19:58:16.697 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.compression: NoCompression 2024-01-21T19:58:16.697 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.bottommost_compression: Disabled 2024-01-21T19:58:16.697 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.prefix_extractor: nullptr 2024-01-21T19:58:16.697 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-21T19:58:16.697 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.num_levels: 7 2024-01-21T19:58:16.697 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-21T19:58:16.697 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-21T19:58:16.697 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-21T19:58:16.697 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-21T19:58:16.697 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-21T19:58:16.697 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-21T19:58:16.697 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-21T19:58:16.697 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-21T19:58:16.697 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-21T19:58:16.697 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-01-21T19:58:16.698 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.compression_opts.level: 32767 2024-01-21T19:58:16.698 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.compression_opts.strategy: 0 2024-01-21T19:58:16.698 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-21T19:58:16.698 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-21T19:58:16.698 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.compression_opts.enabled: false 2024-01-21T19:58:16.698 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-21T19:58:16.698 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-21T19:58:16.698 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-21T19:58:16.698 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.target_file_size_base: 67108864 2024-01-21T19:58:16.698 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-01-21T19:58:16.698 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-21T19:58:16.698 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-21T19:58:16.698 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-21T19:58:16.698 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-21T19:58:16.698 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-21T19:58:16.699 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-21T19:58:16.699 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-21T19:58:16.699 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-21T19:58:16.699 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-21T19:58:16.699 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-21T19:58:16.699 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-21T19:58:16.699 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-21T19:58:16.699 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.arena_block_size: 4194304 2024-01-21T19:58:16.699 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-21T19:58:16.699 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-21T19:58:16.699 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-01-21T19:58:16.699 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.disable_auto_compactions: 0 2024-01-21T19:58:16.699 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-21T19:58:16.699 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-21T19:58:16.699 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-21T19:58:16.700 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-21T19:58:16.700 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-21T19:58:16.701 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-21T19:58:16.701 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-21T19:58:16.701 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-21T19:58:16.701 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-21T19:58:16.701 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-21T19:58:16.701 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.table_properties_collectors: 2024-01-21T19:58:16.701 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.inplace_update_support: 0 2024-01-21T19:58:16.701 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-01-21T19:58:16.701 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-21T19:58:16.701 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-21T19:58:16.702 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-01-21T19:58:16.702 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.bloom_locality: 0 2024-01-21T19:58:16.702 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.max_successive_merges: 0 2024-01-21T19:58:16.702 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-01-21T19:58:16.702 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.paranoid_file_checks: 0 2024-01-21T19:58:16.702 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.force_consistency_checks: 0 2024-01-21T19:58:16.702 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.report_bg_io_stats: 0 2024-01-21T19:58:16.702 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.ttl: 2592000 2024-01-21T19:58:16.702 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.406+0000 7fb6899c9700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-01-21T19:58:16.702 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.407+0000 7fb6899c9700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi053/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-21T19:58:16.702 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: 2024-01-21T19:58:16.702 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.407+0000 7fb6899c9700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2024-01-21T19:58:16.702 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: 2024-01-21T19:58:16.703 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.407+0000 7fb6899c9700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705867096408904, "job": 1, "event": "recovery_started", "log_files": [6]} 2024-01-21T19:58:16.703 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.407+0000 7fb6899c9700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2024-01-21T19:58:16.703 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.408+0000 7fb6899c9700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-01-21T19:58:16.703 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.408+0000 7fb6899c9700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705867096409844, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 88698, "table_properties": {"data_size": 86813, "index_size": 362, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12384, "raw_average_key_size": 49, "raw_value_size": 79954, "raw_average_value_size": 321, "num_data_blocks": 11, "num_entries": 249, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1705867096, "oldest_key_time": 3, "file_creation_time": 0}} 2024-01-21T19:58:16.703 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.408+0000 7fb6899c9700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2024-01-21T19:58:16.703 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: 2024-01-21T19:58:16.703 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.409+0000 7fb6899c9700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705867096410324, "job": 1, "event": "recovery_finished"} 2024-01-21T19:58:16.703 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.409+0000 7fb6899c9700 4 rocksdb: DB pointer 0x563d9bc89800 2024-01-21T19:58:16.703 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.409+0000 7fb6707e1700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-01-21T19:58:16.703 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.409+0000 7fb6707e1700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-01-21T19:58:16.703 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: ** DB Stats ** 2024-01-21T19:58:16.703 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-21T19:58:16.703 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-21T19:58:16.703 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-21T19:58:16.704 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-21T19:58:16.704 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-21T19:58:16.704 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-01-21T19:58:16.704 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-21T19:58:16.704 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: 2024-01-21T19:58:16.704 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: ** Compaction Stats [default] ** 2024-01-21T19:58:16.704 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T19:58:16.704 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T19:58:16.704 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: L0 2/0 88.31 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 139.6 0.00 0.00 1 0.001 0 0 2024-01-21T19:58:16.704 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: Sum 2/0 88.31 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 139.6 0.00 0.00 1 0.001 0 0 2024-01-21T19:58:16.704 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 139.6 0.00 0.00 1 0.001 0 0 2024-01-21T19:58:16.704 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: 2024-01-21T19:58:16.704 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: ** Compaction Stats [default] ** 2024-01-21T19:58:16.704 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T19:58:16.705 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T19:58:16.705 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 139.6 0.00 0.00 1 0.001 0 0 2024-01-21T19:58:16.705 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-21T19:58:16.705 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-21T19:58:16.705 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-21T19:58:16.705 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: AddFile(Total Files): cumulative 0, interval 0 2024-01-21T19:58:16.705 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-21T19:58:16.705 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: AddFile(Keys): cumulative 0, interval 0 2024-01-21T19:58:16.705 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: Cumulative compaction: 0.00 GB write, 27.38 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T19:58:16.705 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: Interval compaction: 0.00 GB write, 27.38 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T19:58:16.705 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-21T19:58:16.705 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: 2024-01-21T19:58:16.705 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: ** File Read Latency Histogram By Level [default] ** 2024-01-21T19:58:16.705 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: 2024-01-21T19:58:16.705 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: ** Compaction Stats [default] ** 2024-01-21T19:58:16.705 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T19:58:16.706 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T19:58:16.706 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: L0 2/0 88.31 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 139.6 0.00 0.00 1 0.001 0 0 2024-01-21T19:58:16.706 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: Sum 2/0 88.31 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 139.6 0.00 0.00 1 0.001 0 0 2024-01-21T19:58:16.706 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-21T19:58:16.706 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: 2024-01-21T19:58:16.706 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: ** Compaction Stats [default] ** 2024-01-21T19:58:16.706 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T19:58:16.706 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T19:58:16.706 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 139.6 0.00 0.00 1 0.001 0 0 2024-01-21T19:58:16.706 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-21T19:58:16.706 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-21T19:58:16.706 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-21T19:58:16.706 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: AddFile(Total Files): cumulative 0, interval 0 2024-01-21T19:58:16.706 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-21T19:58:16.707 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: AddFile(Keys): cumulative 0, interval 0 2024-01-21T19:58:16.707 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: Cumulative compaction: 0.00 GB write, 27.11 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T19:58:16.707 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T19:58:16.707 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-21T19:58:16.707 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: 2024-01-21T19:58:16.707 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: ** File Read Latency Histogram By Level [default] ** 2024-01-21T19:58:16.707 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: 2024-01-21T19:58:16.707 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.410+0000 7fb6899c9700 0 starting mon.smithi053 rank 0 at public addrs [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] at bind addrs [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi053 fsid 57f6722c-b897-11ee-95b0-87774f69a715 2024-01-21T19:58:16.707 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.411+0000 7fb6899c9700 1 mon.smithi053@-1(???) e1 preinit fsid 57f6722c-b897-11ee-95b0-87774f69a715 2024-01-21T19:58:16.707 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.411+0000 7fb6899c9700 0 mon.smithi053@-1(???).mds e1 new map 2024-01-21T19:58:16.707 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.411+0000 7fb6899c9700 0 mon.smithi053@-1(???).mds e1 print_map 2024-01-21T19:58:16.707 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: e1 2024-01-21T19:58:16.707 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: enable_multiple, ever_enabled_multiple: 1,1 2024-01-21T19:58:16.707 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-21T19:58:16.707 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: legacy client fscid: -1 2024-01-21T19:58:16.708 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: 2024-01-21T19:58:16.708 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: No filesystems configured 2024-01-21T19:58:16.708 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: 2024-01-21T19:58:16.708 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.411+0000 7fb6899c9700 0 mon.smithi053@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-01-21T19:58:16.708 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.411+0000 7fb6899c9700 0 mon.smithi053@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-21T19:58:16.708 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.411+0000 7fb6899c9700 0 mon.smithi053@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-21T19:58:16.708 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.411+0000 7fb6899c9700 0 mon.smithi053@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-21T19:58:16.708 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:14.337612+0000 mon.smithi053 (mon.0) 8 : audit [INF] from='client.? 172.21.15.53:0/3866529046' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-21T19:58:16.708 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:14.340675+0000 mon.smithi053 (mon.0) 9 : audit [INF] from='client.? 172.21.15.53:0/3866529046' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-01-21T19:58:16.708 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.412+0000 7fb6899c9700 1 mon.smithi053@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-01-21T19:58:16.708 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.413+0000 7fb6899c9700 20 expand_channel_meta expand map: {default=false} 2024-01-21T19:58:16.708 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.413+0000 7fb6899c9700 20 expand_channel_meta from 'false' to 'false' 2024-01-21T19:58:16.708 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.413+0000 7fb6899c9700 20 expand_channel_meta expanded map: {default=false} 2024-01-21T19:58:16.709 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.413+0000 7fb6899c9700 20 expand_channel_meta expand map: {default=info} 2024-01-21T19:58:16.709 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.413+0000 7fb6899c9700 20 expand_channel_meta from 'info' to 'info' 2024-01-21T19:58:16.709 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.413+0000 7fb6899c9700 20 expand_channel_meta expanded map: {default=info} 2024-01-21T19:58:16.709 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.413+0000 7fb6899c9700 20 expand_channel_meta expand map: {default=daemon} 2024-01-21T19:58:16.709 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.413+0000 7fb6899c9700 20 expand_channel_meta from 'daemon' to 'daemon' 2024-01-21T19:58:16.709 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.413+0000 7fb6899c9700 20 expand_channel_meta expanded map: {default=daemon} 2024-01-21T19:58:16.709 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.413+0000 7fb6899c9700 20 expand_channel_meta expand map: {default=debug} 2024-01-21T19:58:16.709 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.413+0000 7fb6899c9700 20 expand_channel_meta from 'debug' to 'debug' 2024-01-21T19:58:16.709 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T19:58:16.413+0000 7fb6899c9700 20 expand_channel_meta expanded map: {default=debug} 2024-01-21T19:58:16.709 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:58:16.428993+0000 mon.smithi053 (mon.0) 1 : cluster [INF] mon.smithi053 is new leader, mons smithi053 in quorum (ranks 0) 2024-01-21T19:58:16.709 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:58:16.429061+0000 mon.smithi053 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi053=[v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0]} 2024-01-21T19:58:16.709 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:58:16.429117+0000 mon.smithi053 (mon.0) 3 : cluster [DBG] fsmap 2024-01-21T19:58:16.709 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:58:16.429152+0000 mon.smithi053 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-01-21T19:58:16.710 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:58:16.429537+0000 mon.smithi053 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2024-01-21T19:58:16.760 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:16.768+0000 7ff5647fc700 1 Processor -- start 2024-01-21T19:58:16.761 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:16.768+0000 7ff5647fc700 1 -- start start 2024-01-21T19:58:16.761 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:16.768+0000 7ff5647fc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff55c0703d0 0x7ff55c0fe940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:16.761 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:16.768+0000 7ff5647fc700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff55c0fee30 con 0x7ff55c0703d0 2024-01-21T19:58:16.763 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:16.769+0000 7ff562598700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff55c0703d0 0x7ff55c0fe940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:16.763 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:16.769+0000 7ff562598700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff55c0703d0 0x7ff55c0fe940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52998/0 (socket says 172.21.15.53:52998) 2024-01-21T19:58:16.763 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:16.769+0000 7ff562598700 1 -- 172.21.15.53:0/2677369013 learned_addr learned my addr 172.21.15.53:0/2677369013 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:58:16.763 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:16.769+0000 7ff562598700 1 -- 172.21.15.53:0/2677369013 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff55c0fef70 con 0x7ff55c0703d0 2024-01-21T19:58:16.763 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:16.769+0000 7ff562598700 1 --2- 172.21.15.53:0/2677369013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff55c0703d0 0x7ff55c0fe940 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7ff55800b5f0 tx=0x7ff558006ee0).ready entity=mon.0 client_cookie=62b231beb6f88a0c server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:16.763 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:16.769+0000 7ff561596700 1 -- 172.21.15.53:0/2677369013 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff55800d600 con 0x7ff55c0703d0 2024-01-21T19:58:16.763 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:16.770+0000 7ff561596700 1 -- 172.21.15.53:0/2677369013 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff55800dbd0 con 0x7ff55c0703d0 2024-01-21T19:58:16.764 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:16.770+0000 7ff561596700 1 -- 172.21.15.53:0/2677369013 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff558011c00 con 0x7ff55c0703d0 2024-01-21T19:58:16.764 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:16.770+0000 7ff5647fc700 1 -- 172.21.15.53:0/2677369013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff55c0703d0 msgr2=0x7ff55c0fe940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:16.764 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:16.770+0000 7ff5647fc700 1 --2- 172.21.15.53:0/2677369013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff55c0703d0 0x7ff55c0fe940 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7ff55800b5f0 tx=0x7ff558006ee0).stop 2024-01-21T19:58:16.764 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:16.770+0000 7ff5647fc700 1 -- 172.21.15.53:0/2677369013 shutdown_connections 2024-01-21T19:58:16.764 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:16.770+0000 7ff5647fc700 1 --2- 172.21.15.53:0/2677369013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff55c0703d0 0x7ff55c0fe940 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:16.764 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:16.770+0000 7ff5647fc700 1 -- 172.21.15.53:0/2677369013 >> 172.21.15.53:0/2677369013 conn(0x7ff55c0f63c0 msgr2=0x7ff55c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:58:16.764 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:16.770+0000 7ff5647fc700 1 -- 172.21.15.53:0/2677369013 shutdown_connections 2024-01-21T19:58:16.764 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:16.770+0000 7ff5647fc700 1 -- 172.21.15.53:0/2677369013 wait complete. 2024-01-21T19:58:16.764 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:16.771+0000 7ff5647fc700 1 Processor -- start 2024-01-21T19:58:16.764 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:16.771+0000 7ff5647fc700 1 -- start start 2024-01-21T19:58:16.764 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:16.771+0000 7ff5647fc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff55c0703d0 0x7ff55c0fc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:16.764 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:16.771+0000 7ff5647fc700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff55c0fa840 con 0x7ff55c0703d0 2024-01-21T19:58:16.764 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:16.771+0000 7ff562598700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff55c0703d0 0x7ff55c0fc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:16.764 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:16.771+0000 7ff562598700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff55c0703d0 0x7ff55c0fc3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:53006/0 (socket says 172.21.15.53:53006) 2024-01-21T19:58:16.765 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:16.771+0000 7ff562598700 1 -- 172.21.15.53:0/2987589048 learned_addr learned my addr 172.21.15.53:0/2987589048 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:58:16.765 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:16.771+0000 7ff562598700 1 -- 172.21.15.53:0/2987589048 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff55800b200 con 0x7ff55c0703d0 2024-01-21T19:58:16.765 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:16.771+0000 7ff562598700 1 --2- 172.21.15.53:0/2987589048 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff55c0703d0 0x7ff55c0fc3c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7ff55800e040 tx=0x7ff558006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:16.765 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:16.772+0000 7ff54affd700 1 -- 172.21.15.53:0/2987589048 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff55800d9f0 con 0x7ff55c0703d0 2024-01-21T19:58:16.765 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:16.772+0000 7ff54affd700 1 -- 172.21.15.53:0/2987589048 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff55801a9f0 con 0x7ff55c0703d0 2024-01-21T19:58:16.765 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:16.772+0000 7ff5647fc700 1 -- 172.21.15.53:0/2987589048 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff55c0fa9e0 con 0x7ff55c0703d0 2024-01-21T19:58:16.765 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:16.772+0000 7ff54affd700 1 -- 172.21.15.53:0/2987589048 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff55801b8c0 con 0x7ff55c0703d0 2024-01-21T19:58:16.766 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:16.772+0000 7ff5647fc700 1 -- 172.21.15.53:0/2987589048 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff55c0fadf0 con 0x7ff55c0703d0 2024-01-21T19:58:16.766 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:16.772+0000 7ff54affd700 1 -- 172.21.15.53:0/2987589048 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7ff55800f070 con 0x7ff55c0703d0 2024-01-21T19:58:16.766 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:16.772+0000 7ff54affd700 1 -- 172.21.15.53:0/2987589048 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7ff558025b30 con 0x7ff55c0703d0 2024-01-21T19:58:16.766 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:16.773+0000 7ff5647fc700 1 -- 172.21.15.53:0/2987589048 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff55c045ad0 con 0x7ff55c0703d0 2024-01-21T19:58:16.767 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:16.775+0000 7ff54affd700 1 -- 172.21.15.53:0/2987589048 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7ff558016070 con 0x7ff55c0703d0 2024-01-21T19:58:16.833 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:16.841+0000 7ff5647fc700 1 -- 172.21.15.53:0/2987589048 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7ff55c0ffb90 con 0x7ff55c0703d0 2024-01-21T19:58:16.835 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:16.843+0000 7ff54affd700 1 -- 172.21.15.53:0/2987589048 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7ff55801bd70 con 0x7ff55c0703d0 2024-01-21T19:58:16.836 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:16.844+0000 7ff5647fc700 1 -- 172.21.15.53:0/2987589048 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff55c0703d0 msgr2=0x7ff55c0fc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:16.836 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:16.844+0000 7ff5647fc700 1 --2- 172.21.15.53:0/2987589048 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff55c0703d0 0x7ff55c0fc3c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7ff55800e040 tx=0x7ff558006ee0).stop 2024-01-21T19:58:16.837 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:16.844+0000 7ff5647fc700 1 -- 172.21.15.53:0/2987589048 shutdown_connections 2024-01-21T19:58:16.837 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:16.844+0000 7ff5647fc700 1 --2- 172.21.15.53:0/2987589048 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff55c0703d0 0x7ff55c0fc3c0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:16.837 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:16.844+0000 7ff5647fc700 1 -- 172.21.15.53:0/2987589048 >> 172.21.15.53:0/2987589048 conn(0x7ff55c0f63c0 msgr2=0x7ff55c0f7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:58:16.837 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:16.844+0000 7ff5647fc700 1 -- 172.21.15.53:0/2987589048 shutdown_connections 2024-01-21T19:58:16.837 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:16.844+0000 7ff5647fc700 1 -- 172.21.15.53:0/2987589048 wait complete. 2024-01-21T19:58:17.229 INFO:teuthology.orchestra.run.smithi053.stderr:Wrote config to /etc/ceph/ceph.conf 2024-01-21T19:58:17.229 INFO:teuthology.orchestra.run.smithi053.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-01-21T19:58:17.229 INFO:teuthology.orchestra.run.smithi053.stderr:Creating mgr... 2024-01-21T19:58:17.229 INFO:teuthology.orchestra.run.smithi053.stderr:Verifying port 9283 ... 2024-01-21T19:58:17.231 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl daemon-reload 2024-01-21T19:58:17.331 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl stop ceph-57f6722c-b897-11ee-95b0-87774f69a715@mgr.smithi053.fgsmol 2024-01-21T19:58:17.356 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl reset-failed ceph-57f6722c-b897-11ee-95b0-87774f69a715@mgr.smithi053.fgsmol 2024-01-21T19:58:17.361 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: Failed to reset failed state of unit ceph-57f6722c-b897-11ee-95b0-87774f69a715@mgr.smithi053.fgsmol.service: Unit ceph-57f6722c-b897-11ee-95b0-87774f69a715@mgr.smithi053.fgsmol.service not loaded. 2024-01-21T19:58:17.362 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl enable ceph-57f6722c-b897-11ee-95b0-87774f69a715@mgr.smithi053.fgsmol 2024-01-21T19:58:17.368 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: Created symlink /etc/systemd/system/ceph-57f6722c-b897-11ee-95b0-87774f69a715.target.wants/ceph-57f6722c-b897-11ee-95b0-87774f69a715@mgr.smithi053.fgsmol.service → /etc/systemd/system/ceph-57f6722c-b897-11ee-95b0-87774f69a715@.service. 2024-01-21T19:58:17.460 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl start ceph-57f6722c-b897-11ee-95b0-87774f69a715@mgr.smithi053.fgsmol 2024-01-21T19:58:17.805 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-21T19:58:17.815 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: disabled 2024-01-21T19:58:17.815 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl is-active firewalld.service 2024-01-21T19:58:17.823 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: inactive 2024-01-21T19:58:17.824 INFO:teuthology.orchestra.run.smithi053.stderr:firewalld.service is not enabled 2024-01-21T19:58:17.824 INFO:teuthology.orchestra.run.smithi053.stderr:Not possible to enable service . firewalld.service is not available 2024-01-21T19:58:17.824 INFO:teuthology.orchestra.run.smithi053.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-01-21T19:58:17.824 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-21T19:58:17.831 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: disabled 2024-01-21T19:58:17.832 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl is-active firewalld.service 2024-01-21T19:58:17.840 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: inactive 2024-01-21T19:58:17.841 INFO:teuthology.orchestra.run.smithi053.stderr:firewalld.service is not enabled 2024-01-21T19:58:17.841 INFO:teuthology.orchestra.run.smithi053.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-01-21T19:58:17.841 INFO:teuthology.orchestra.run.smithi053.stderr:Waiting for mgr to start... 2024-01-21T19:58:17.841 INFO:teuthology.orchestra.run.smithi053.stderr:Waiting for mgr... 2024-01-21T19:58:17.841 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/57f6722c-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpw7bt2for:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp987f45g0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-01-21T19:58:18.184 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:17 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:16.844966+0000 mon.smithi053 (mon.0) 6 : audit [INF] from='client.? 172.21.15.53:0/2987589048' entity='client.admin' 2024-01-21T19:58:18.198 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:18.206+0000 7f5cff59e700 1 Processor -- start 2024-01-21T19:58:18.200 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:18.206+0000 7f5cff59e700 1 -- start start 2024-01-21T19:58:18.200 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:18.206+0000 7f5cff59e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d000724b0 0x7f5d00070a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:18.200 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:18.206+0000 7f5cff59e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d000728d0 con 0x7f5d000724b0 2024-01-21T19:58:18.200 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:18.207+0000 7f5cfe59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d000724b0 0x7f5d00070a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:18.200 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:18.207+0000 7f5cfe59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d000724b0 0x7f5d00070a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:53038/0 (socket says 172.21.15.53:53038) 2024-01-21T19:58:18.200 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:18.207+0000 7f5cfe59c700 1 -- 172.21.15.53:0/1188869776 learned_addr learned my addr 172.21.15.53:0/1188869776 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:58:18.200 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:18.207+0000 7f5cfe59c700 1 -- 172.21.15.53:0/1188869776 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d00070f40 con 0x7f5d000724b0 2024-01-21T19:58:18.201 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:18.207+0000 7f5cfe59c700 1 --2- 172.21.15.53:0/1188869776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d000724b0 0x7f5d00070a50 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f5cf000b5f0 tx=0x7f5cf0006ee0).ready entity=mon.0 client_cookie=1265d733ee90258e server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:18.201 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:18.207+0000 7f5cfd59a700 1 -- 172.21.15.53:0/1188869776 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cf000d600 con 0x7f5d000724b0 2024-01-21T19:58:18.201 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:18.207+0000 7f5cfd59a700 1 -- 172.21.15.53:0/1188869776 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f5cf000dbd0 con 0x7f5d000724b0 2024-01-21T19:58:18.201 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:18.207+0000 7f5cfd59a700 1 -- 172.21.15.53:0/1188869776 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cf0011c00 con 0x7f5d000724b0 2024-01-21T19:58:18.201 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:18.208+0000 7f5cff59e700 1 -- 172.21.15.53:0/1188869776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d000724b0 msgr2=0x7f5d00070a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:18.201 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:18.208+0000 7f5cff59e700 1 --2- 172.21.15.53:0/1188869776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d000724b0 0x7f5d00070a50 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f5cf000b5f0 tx=0x7f5cf0006ee0).stop 2024-01-21T19:58:18.201 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:18.208+0000 7f5cff59e700 1 -- 172.21.15.53:0/1188869776 shutdown_connections 2024-01-21T19:58:18.201 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:18.208+0000 7f5cff59e700 1 --2- 172.21.15.53:0/1188869776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d000724b0 0x7f5d00070a50 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:18.201 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:18.208+0000 7f5cff59e700 1 -- 172.21.15.53:0/1188869776 >> 172.21.15.53:0/1188869776 conn(0x7f5d000f7380 msgr2=0x7f5d000f97a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:58:18.201 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:18.208+0000 7f5cff59e700 1 -- 172.21.15.53:0/1188869776 shutdown_connections 2024-01-21T19:58:18.201 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:18.208+0000 7f5cff59e700 1 -- 172.21.15.53:0/1188869776 wait complete. 2024-01-21T19:58:18.203 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:18.209+0000 7f5cff59e700 1 Processor -- start 2024-01-21T19:58:18.203 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:18.209+0000 7f5cff59e700 1 -- start start 2024-01-21T19:58:18.203 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:18.209+0000 7f5cff59e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d000724b0 0x7f5d0010cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:18.203 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:18.209+0000 7f5cff59e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0010d010 con 0x7f5d000724b0 2024-01-21T19:58:18.203 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:18.209+0000 7f5cfe59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d000724b0 0x7f5d0010cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:18.203 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:18.209+0000 7f5cfe59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d000724b0 0x7f5d0010cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:53052/0 (socket says 172.21.15.53:53052) 2024-01-21T19:58:18.203 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:18.209+0000 7f5cfe59c700 1 -- 172.21.15.53:0/2778580724 learned_addr learned my addr 172.21.15.53:0/2778580724 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:58:18.203 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:18.209+0000 7f5cfe59c700 1 -- 172.21.15.53:0/2778580724 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cf000b200 con 0x7f5d000724b0 2024-01-21T19:58:18.203 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:18.210+0000 7f5cfe59c700 1 --2- 172.21.15.53:0/2778580724 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d000724b0 0x7f5d0010cb20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f5cf0002f40 tx=0x7f5cf0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:18.204 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:18.210+0000 7f5ceeffd700 1 -- 172.21.15.53:0/2778580724 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cf0011690 con 0x7f5d000724b0 2024-01-21T19:58:18.204 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:18.210+0000 7f5ceeffd700 1 -- 172.21.15.53:0/2778580724 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f5cf001b880 con 0x7f5d000724b0 2024-01-21T19:58:18.204 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:18.210+0000 7f5cff59e700 1 -- 172.21.15.53:0/2778580724 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d0010d210 con 0x7f5d000724b0 2024-01-21T19:58:18.204 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:18.210+0000 7f5ceeffd700 1 -- 172.21.15.53:0/2778580724 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cf001a6b0 con 0x7f5d000724b0 2024-01-21T19:58:18.204 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:18.210+0000 7f5cff59e700 1 -- 172.21.15.53:0/2778580724 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d0010d620 con 0x7f5d000724b0 2024-01-21T19:58:18.204 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:18.210+0000 7f5ceeffd700 1 -- 172.21.15.53:0/2778580724 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f5cf000f070 con 0x7f5d000724b0 2024-01-21T19:58:18.204 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:18.210+0000 7f5ceeffd700 1 -- 172.21.15.53:0/2778580724 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f5cf00259f0 con 0x7f5d000724b0 2024-01-21T19:58:18.204 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:18.211+0000 7f5cff59e700 1 -- 172.21.15.53:0/2778580724 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d00045ad0 con 0x7f5d000724b0 2024-01-21T19:58:18.205 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:18.213+0000 7f5ceeffd700 1 -- 172.21.15.53:0/2778580724 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f5cf0016070 con 0x7f5d000724b0 2024-01-21T19:58:18.274 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:18.282+0000 7f5cff59e700 1 -- 172.21.15.53:0/2778580724 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f5d00106880 con 0x7f5d000724b0 2024-01-21T19:58:18.276 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:18.283+0000 7f5ceeffd700 1 -- 172.21.15.53:0/2778580724 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f5cf001bd60 con 0x7f5d000724b0 2024-01-21T19:58:18.276 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:18.276 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: { 2024-01-21T19:58:18.276 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "fsid": "57f6722c-b897-11ee-95b0-87774f69a715", 2024-01-21T19:58:18.276 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "health": { 2024-01-21T19:58:18.276 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-21T19:58:18.276 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "checks": {}, 2024-01-21T19:58:18.276 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "mutes": [] 2024-01-21T19:58:18.276 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-01-21T19:58:18.276 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-21T19:58:18.276 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "quorum": [ 2024-01-21T19:58:18.282 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 0 2024-01-21T19:58:18.282 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: ], 2024-01-21T19:58:18.282 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-21T19:58:18.282 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "smithi053" 2024-01-21T19:58:18.282 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: ], 2024-01-21T19:58:18.282 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "quorum_age": 1, 2024-01-21T19:58:18.282 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "monmap": { 2024-01-21T19:58:18.282 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T19:58:18.282 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-21T19:58:18.282 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-21T19:58:18.283 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-01-21T19:58:18.283 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "osdmap": { 2024-01-21T19:58:18.283 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T19:58:18.283 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-21T19:58:18.283 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-21T19:58:18.283 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-21T19:58:18.283 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-21T19:58:18.283 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-21T19:58:18.283 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-21T19:58:18.283 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-01-21T19:58:18.283 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "pgmap": { 2024-01-21T19:58:18.283 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-21T19:58:18.283 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-21T19:58:18.283 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-21T19:58:18.283 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-21T19:58:18.283 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-21T19:58:18.284 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-21T19:58:18.284 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-21T19:58:18.284 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-21T19:58:18.284 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-01-21T19:58:18.284 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "fsmap": { 2024-01-21T19:58:18.284 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T19:58:18.284 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-21T19:58:18.284 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-21T19:58:18.284 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-01-21T19:58:18.284 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-21T19:58:18.284 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "available": false, 2024-01-21T19:58:18.284 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-21T19:58:18.284 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "modules": [ 2024-01-21T19:58:18.284 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "iostat", 2024-01-21T19:58:18.284 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "restful" 2024-01-21T19:58:18.285 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: ], 2024-01-21T19:58:18.285 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "services": {} 2024-01-21T19:58:18.285 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-01-21T19:58:18.285 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "servicemap": { 2024-01-21T19:58:18.285 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T19:58:18.285 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "modified": "2024-01-21T19:58:13.005835+0000", 2024-01-21T19:58:18.285 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "services": {} 2024-01-21T19:58:18.285 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-01-21T19:58:18.285 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-21T19:58:18.285 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: } 2024-01-21T19:58:18.285 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:18.284+0000 7f5cff59e700 1 -- 172.21.15.53:0/2778580724 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d000724b0 msgr2=0x7f5d0010cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:18.285 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:18.284+0000 7f5cff59e700 1 --2- 172.21.15.53:0/2778580724 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d000724b0 0x7f5d0010cb20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f5cf0002f40 tx=0x7f5cf0006ee0).stop 2024-01-21T19:58:18.285 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:18.285+0000 7f5cff59e700 1 -- 172.21.15.53:0/2778580724 shutdown_connections 2024-01-21T19:58:18.285 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:18.285+0000 7f5cff59e700 1 --2- 172.21.15.53:0/2778580724 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d000724b0 0x7f5d0010cb20 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:18.285 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:18.285+0000 7f5cff59e700 1 -- 172.21.15.53:0/2778580724 >> 172.21.15.53:0/2778580724 conn(0x7f5d000f7380 msgr2=0x7f5d000f8fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:58:18.285 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:18.285+0000 7f5cff59e700 1 -- 172.21.15.53:0/2778580724 shutdown_connections 2024-01-21T19:58:18.285 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:18.285+0000 7f5cff59e700 1 -- 172.21.15.53:0/2778580724 wait complete. 2024-01-21T19:58:18.687 INFO:teuthology.orchestra.run.smithi053.stderr:mgr not available, waiting (1/15)... 2024-01-21T19:58:18.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:18 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:18.284419+0000 mon.smithi053 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.53:0/2778580724' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-21T19:58:20.689 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/57f6722c-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpw7bt2for:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp987f45g0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-01-21T19:58:21.038 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:21.044+0000 7f6040c0b700 1 Processor -- start 2024-01-21T19:58:21.038 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:21.044+0000 7f6040c0b700 1 -- start start 2024-01-21T19:58:21.039 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:21.044+0000 7f6040c0b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f603c070430 0x7f603c0fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:21.039 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:21.044+0000 7f6040c0b700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603c0fedd0 con 0x7f603c070430 2024-01-21T19:58:21.039 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:21.045+0000 7f603a59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f603c070430 0x7f603c0fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:21.039 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:21.045+0000 7f603a59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f603c070430 0x7f603c0fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:53056/0 (socket says 172.21.15.53:53056) 2024-01-21T19:58:21.039 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:21.045+0000 7f603a59c700 1 -- 172.21.15.53:0/2091679787 learned_addr learned my addr 172.21.15.53:0/2091679787 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:58:21.039 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:21.045+0000 7f603a59c700 1 -- 172.21.15.53:0/2091679787 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f603c0fef10 con 0x7f603c070430 2024-01-21T19:58:21.039 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:21.045+0000 7f603a59c700 1 --2- 172.21.15.53:0/2091679787 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f603c070430 0x7f603c0fe8e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f602c00b5f0 tx=0x7f602c006ee0).ready entity=mon.0 client_cookie=3e1cd5a235a8ab7c server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:21.039 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:21.045+0000 7f603959a700 1 -- 172.21.15.53:0/2091679787 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f602c00d600 con 0x7f603c070430 2024-01-21T19:58:21.039 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:21.045+0000 7f603959a700 1 -- 172.21.15.53:0/2091679787 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f602c00dbd0 con 0x7f603c070430 2024-01-21T19:58:21.039 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:21.045+0000 7f603959a700 1 -- 172.21.15.53:0/2091679787 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f602c011c00 con 0x7f603c070430 2024-01-21T19:58:21.039 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:21.046+0000 7f6040c0b700 1 -- 172.21.15.53:0/2091679787 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f603c070430 msgr2=0x7f603c0fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:21.039 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:21.046+0000 7f6040c0b700 1 --2- 172.21.15.53:0/2091679787 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f603c070430 0x7f603c0fe8e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f602c00b5f0 tx=0x7f602c006ee0).stop 2024-01-21T19:58:21.039 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:21.046+0000 7f6040c0b700 1 -- 172.21.15.53:0/2091679787 shutdown_connections 2024-01-21T19:58:21.040 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:21.046+0000 7f6040c0b700 1 --2- 172.21.15.53:0/2091679787 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f603c070430 0x7f603c0fe8e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:21.040 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:21.046+0000 7f6040c0b700 1 -- 172.21.15.53:0/2091679787 >> 172.21.15.53:0/2091679787 conn(0x7f603c0f63c0 msgr2=0x7f603c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:58:21.040 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:21.046+0000 7f6040c0b700 1 -- 172.21.15.53:0/2091679787 shutdown_connections 2024-01-21T19:58:21.040 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:21.046+0000 7f6040c0b700 1 -- 172.21.15.53:0/2091679787 wait complete. 2024-01-21T19:58:21.041 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:21.047+0000 7f6040c0b700 1 Processor -- start 2024-01-21T19:58:21.041 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:21.047+0000 7f6040c0b700 1 -- start start 2024-01-21T19:58:21.041 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:21.047+0000 7f6040c0b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f603c070430 0x7f603c10aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:21.041 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:21.047+0000 7f6040c0b700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603c10aef0 con 0x7f603c070430 2024-01-21T19:58:21.041 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:21.047+0000 7f603a59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f603c070430 0x7f603c10aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:21.041 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:21.047+0000 7f603a59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f603c070430 0x7f603c10aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:53068/0 (socket says 172.21.15.53:53068) 2024-01-21T19:58:21.041 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:21.048+0000 7f603a59c700 1 -- 172.21.15.53:0/1299936437 learned_addr learned my addr 172.21.15.53:0/1299936437 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:58:21.041 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:21.048+0000 7f603a59c700 1 -- 172.21.15.53:0/1299936437 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f602c00b200 con 0x7f603c070430 2024-01-21T19:58:21.042 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:21.048+0000 7f603a59c700 1 --2- 172.21.15.53:0/1299936437 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f603c070430 0x7f603c10aa00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f602c00b190 tx=0x7f602c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:21.042 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:21.048+0000 7f602affd700 1 -- 172.21.15.53:0/1299936437 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f602c01b410 con 0x7f603c070430 2024-01-21T19:58:21.042 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:21.048+0000 7f602affd700 1 -- 172.21.15.53:0/1299936437 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f602c01b9e0 con 0x7f603c070430 2024-01-21T19:58:21.042 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:21.048+0000 7f602affd700 1 -- 172.21.15.53:0/1299936437 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f602c01a880 con 0x7f603c070430 2024-01-21T19:58:21.042 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:21.048+0000 7f6040c0b700 1 -- 172.21.15.53:0/1299936437 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f603c10b0f0 con 0x7f603c070430 2024-01-21T19:58:21.042 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:21.048+0000 7f6040c0b700 1 -- 172.21.15.53:0/1299936437 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f603c10b500 con 0x7f603c070430 2024-01-21T19:58:21.042 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:21.049+0000 7f602affd700 1 -- 172.21.15.53:0/1299936437 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f602c00f070 con 0x7f603c070430 2024-01-21T19:58:21.042 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:21.049+0000 7f602affd700 1 -- 172.21.15.53:0/1299936437 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f602c025c90 con 0x7f603c070430 2024-01-21T19:58:21.042 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:21.049+0000 7f6040c0b700 1 -- 172.21.15.53:0/1299936437 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f603c045ad0 con 0x7f603c070430 2024-01-21T19:58:21.043 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:21.051+0000 7f602affd700 1 -- 172.21.15.53:0/1299936437 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f602c016070 con 0x7f603c070430 2024-01-21T19:58:21.114 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:21.121+0000 7f6040c0b700 1 -- 172.21.15.53:0/1299936437 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f603c1047f0 con 0x7f603c070430 2024-01-21T19:58:21.114 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:21.122+0000 7f602affd700 1 -- 172.21.15.53:0/1299936437 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f602c01ae20 con 0x7f603c070430 2024-01-21T19:58:21.114 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:21.115 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: { 2024-01-21T19:58:21.115 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "fsid": "57f6722c-b897-11ee-95b0-87774f69a715", 2024-01-21T19:58:21.115 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "health": { 2024-01-21T19:58:21.115 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-21T19:58:21.115 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "checks": {}, 2024-01-21T19:58:21.115 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "mutes": [] 2024-01-21T19:58:21.115 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-01-21T19:58:21.115 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-21T19:58:21.115 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "quorum": [ 2024-01-21T19:58:21.121 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 0 2024-01-21T19:58:21.121 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: ], 2024-01-21T19:58:21.121 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-21T19:58:21.121 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "smithi053" 2024-01-21T19:58:21.121 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: ], 2024-01-21T19:58:21.121 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "quorum_age": 4, 2024-01-21T19:58:21.121 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "monmap": { 2024-01-21T19:58:21.121 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T19:58:21.121 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-21T19:58:21.121 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-21T19:58:21.121 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-01-21T19:58:21.121 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "osdmap": { 2024-01-21T19:58:21.122 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T19:58:21.122 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-21T19:58:21.122 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-21T19:58:21.122 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-21T19:58:21.122 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-21T19:58:21.122 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-21T19:58:21.122 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-21T19:58:21.122 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-01-21T19:58:21.122 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "pgmap": { 2024-01-21T19:58:21.122 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-21T19:58:21.122 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-21T19:58:21.122 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-21T19:58:21.122 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-21T19:58:21.122 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-21T19:58:21.122 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-21T19:58:21.122 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-21T19:58:21.122 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-21T19:58:21.123 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-01-21T19:58:21.123 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "fsmap": { 2024-01-21T19:58:21.123 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T19:58:21.123 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-21T19:58:21.123 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-21T19:58:21.123 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-01-21T19:58:21.123 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-21T19:58:21.123 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "available": false, 2024-01-21T19:58:21.123 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-21T19:58:21.123 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "modules": [ 2024-01-21T19:58:21.123 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "iostat", 2024-01-21T19:58:21.123 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "restful" 2024-01-21T19:58:21.123 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: ], 2024-01-21T19:58:21.123 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "services": {} 2024-01-21T19:58:21.123 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-01-21T19:58:21.123 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "servicemap": { 2024-01-21T19:58:21.124 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T19:58:21.124 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "modified": "2024-01-21T19:58:13.005835+0000", 2024-01-21T19:58:21.124 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "services": {} 2024-01-21T19:58:21.124 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-01-21T19:58:21.124 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-21T19:58:21.124 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: } 2024-01-21T19:58:21.124 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:21.122+0000 7f6040c0b700 1 -- 172.21.15.53:0/1299936437 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f603c070430 msgr2=0x7f603c10aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:21.124 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:21.122+0000 7f6040c0b700 1 --2- 172.21.15.53:0/1299936437 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f603c070430 0x7f603c10aa00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f602c00b190 tx=0x7f602c006ee0).stop 2024-01-21T19:58:21.124 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:21.122+0000 7f6040c0b700 1 -- 172.21.15.53:0/1299936437 shutdown_connections 2024-01-21T19:58:21.124 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:21.123+0000 7f6040c0b700 1 --2- 172.21.15.53:0/1299936437 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f603c070430 0x7f603c10aa00 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:21.125 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:21.123+0000 7f6040c0b700 1 -- 172.21.15.53:0/1299936437 >> 172.21.15.53:0/1299936437 conn(0x7f603c0f63c0 msgr2=0x7f603c0f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:58:21.125 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:21.123+0000 7f6040c0b700 1 -- 172.21.15.53:0/1299936437 shutdown_connections 2024-01-21T19:58:21.125 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:21.123+0000 7f6040c0b700 1 -- 172.21.15.53:0/1299936437 wait complete. 2024-01-21T19:58:21.388 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:21 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:21.123100+0000 mon.smithi053 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.53:0/1299936437' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-21T19:58:21.470 INFO:teuthology.orchestra.run.smithi053.stderr:mgr not available, waiting (2/15)... 2024-01-21T19:58:23.434 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:23 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:58:22.998128+0000 mon.smithi053 (mon.0) 9 : cluster [INF] Activating manager daemon smithi053.fgsmol 2024-01-21T19:58:23.434 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:23 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:58:23.000561+0000 mon.smithi053 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi053.fgsmol(active, starting, since 0.00255382s) 2024-01-21T19:58:23.434 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:23 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:23.001565+0000 mon.smithi053 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.53:0/1993550010' entity='mgr.smithi053.fgsmol' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-21T19:58:23.434 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:23 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:23.001751+0000 mon.smithi053 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.53:0/1993550010' entity='mgr.smithi053.fgsmol' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-21T19:58:23.434 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:23 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:23.001924+0000 mon.smithi053 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.53:0/1993550010' entity='mgr.smithi053.fgsmol' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-21T19:58:23.435 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:23 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:23.002100+0000 mon.smithi053 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.53:0/1993550010' entity='mgr.smithi053.fgsmol' cmd=[{"prefix": "mon metadata", "id": "smithi053"}]: dispatch 2024-01-21T19:58:23.435 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:23 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:23.002299+0000 mon.smithi053 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.53:0/1993550010' entity='mgr.smithi053.fgsmol' cmd=[{"prefix": "mgr metadata", "who": "smithi053.fgsmol", "id": "smithi053.fgsmol"}]: dispatch 2024-01-21T19:58:23.435 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:23 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:58:23.004023+0000 mon.smithi053 (mon.0) 16 : cluster [INF] Manager daemon smithi053.fgsmol is now available 2024-01-21T19:58:23.435 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:23 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:23.011704+0000 mon.smithi053 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.53:0/1993550010' entity='mgr.smithi053.fgsmol' 2024-01-21T19:58:23.435 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:23 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:23.016341+0000 mon.smithi053 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.53:0/1993550010' entity='mgr.smithi053.fgsmol' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fgsmol/mirror_snapshot_schedule"}]: dispatch 2024-01-21T19:58:23.435 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:23 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:23.017505+0000 mon.smithi053 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.53:0/1993550010' entity='mgr.smithi053.fgsmol' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fgsmol/trash_purge_schedule"}]: dispatch 2024-01-21T19:58:23.435 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:23 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:23.022054+0000 mon.smithi053 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.53:0/1993550010' entity='mgr.smithi053.fgsmol' 2024-01-21T19:58:23.435 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:23 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:23.023819+0000 mon.smithi053 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.53:0/1993550010' entity='mgr.smithi053.fgsmol' 2024-01-21T19:58:23.472 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/57f6722c-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpw7bt2for:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp987f45g0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-01-21T19:58:23.830 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:23.836+0000 7fbcc1a8e700 1 Processor -- start 2024-01-21T19:58:23.830 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:23.836+0000 7fbcc1a8e700 1 -- start start 2024-01-21T19:58:23.830 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:23.837+0000 7fbcc1a8e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbcbc070390 0x7fbcbc100d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:23.830 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:23.837+0000 7fbcc1a8e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcbc0fc770 con 0x7fbcbc070390 2024-01-21T19:58:23.830 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:23.837+0000 7fbcbb7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbcbc070390 0x7fbcbc100d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:23.830 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:23.837+0000 7fbcbb7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbcbc070390 0x7fbcbc100d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:53128/0 (socket says 172.21.15.53:53128) 2024-01-21T19:58:23.831 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:23.837+0000 7fbcbb7fe700 1 -- 172.21.15.53:0/2267437759 learned_addr learned my addr 172.21.15.53:0/2267437759 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:58:23.831 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:23.837+0000 7fbcbb7fe700 1 -- 172.21.15.53:0/2267437759 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcbc101270 con 0x7fbcbc070390 2024-01-21T19:58:23.831 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:23.838+0000 7fbcbb7fe700 1 --2- 172.21.15.53:0/2267437759 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbcbc070390 0x7fbcbc100d80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fbcac00b5f0 tx=0x7fbcac006ee0).ready entity=mon.0 client_cookie=eee504063f4602ea server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:23.831 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:23.838+0000 7fbcba7fc700 1 -- 172.21.15.53:0/2267437759 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcac00d600 con 0x7fbcbc070390 2024-01-21T19:58:23.831 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:23.838+0000 7fbcba7fc700 1 -- 172.21.15.53:0/2267437759 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fbcac00dbd0 con 0x7fbcbc070390 2024-01-21T19:58:23.831 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:23.838+0000 7fbcba7fc700 1 -- 172.21.15.53:0/2267437759 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcac011c00 con 0x7fbcbc070390 2024-01-21T19:58:23.831 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:23.838+0000 7fbcc1a8e700 1 -- 172.21.15.53:0/2267437759 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbcbc070390 msgr2=0x7fbcbc100d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:23.831 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:23.838+0000 7fbcc1a8e700 1 --2- 172.21.15.53:0/2267437759 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbcbc070390 0x7fbcbc100d80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fbcac00b5f0 tx=0x7fbcac006ee0).stop 2024-01-21T19:58:23.834 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:23.838+0000 7fbcc1a8e700 1 -- 172.21.15.53:0/2267437759 shutdown_connections 2024-01-21T19:58:23.834 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:23.838+0000 7fbcc1a8e700 1 --2- 172.21.15.53:0/2267437759 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbcbc070390 0x7fbcbc100d80 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:23.834 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:23.838+0000 7fbcc1a8e700 1 -- 172.21.15.53:0/2267437759 >> 172.21.15.53:0/2267437759 conn(0x7fbcbc0f63c0 msgr2=0x7fbcbc0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:58:23.834 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:23.838+0000 7fbcc1a8e700 1 -- 172.21.15.53:0/2267437759 shutdown_connections 2024-01-21T19:58:23.834 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:23.838+0000 7fbcc1a8e700 1 -- 172.21.15.53:0/2267437759 wait complete. 2024-01-21T19:58:23.834 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:23.839+0000 7fbcc1a8e700 1 Processor -- start 2024-01-21T19:58:23.834 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:23.839+0000 7fbcc1a8e700 1 -- start start 2024-01-21T19:58:23.834 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:23.840+0000 7fbcc1a8e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbcbc070390 0x7fbcbc10ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:23.834 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:23.840+0000 7fbcc1a8e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcbc10d1b0 con 0x7fbcbc070390 2024-01-21T19:58:23.834 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:23.840+0000 7fbcbb7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbcbc070390 0x7fbcbc10ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:23.834 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:23.840+0000 7fbcbb7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbcbc070390 0x7fbcbc10ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:53144/0 (socket says 172.21.15.53:53144) 2024-01-21T19:58:23.834 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:23.840+0000 7fbcbb7fe700 1 -- 172.21.15.53:0/241160857 learned_addr learned my addr 172.21.15.53:0/241160857 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:58:23.834 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:23.840+0000 7fbcbb7fe700 1 -- 172.21.15.53:0/241160857 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcac00b200 con 0x7fbcbc070390 2024-01-21T19:58:23.835 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:23.840+0000 7fbcbb7fe700 1 --2- 172.21.15.53:0/241160857 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbcbc070390 0x7fbcbc10ccc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fbcac00b190 tx=0x7fbcac006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:23.835 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:23.840+0000 7fbcb3fff700 1 -- 172.21.15.53:0/241160857 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcac01b410 con 0x7fbcbc070390 2024-01-21T19:58:23.835 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:23.841+0000 7fbcb3fff700 1 -- 172.21.15.53:0/241160857 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fbcac01b9e0 con 0x7fbcbc070390 2024-01-21T19:58:23.835 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:23.841+0000 7fbcc1a8e700 1 -- 172.21.15.53:0/241160857 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcbc10d3b0 con 0x7fbcbc070390 2024-01-21T19:58:23.835 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:23.841+0000 7fbcb3fff700 1 -- 172.21.15.53:0/241160857 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcac01a880 con 0x7fbcbc070390 2024-01-21T19:58:23.835 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:23.841+0000 7fbcc1a8e700 1 -- 172.21.15.53:0/241160857 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcbc10d7c0 con 0x7fbcbc070390 2024-01-21T19:58:23.835 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:23.841+0000 7fbcb3fff700 1 -- 172.21.15.53:0/241160857 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 2) v1 ==== 35959+0+0 (secure 0 0 0) 0x7fbcac00f070 con 0x7fbcbc070390 2024-01-21T19:58:23.835 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:23.841+0000 7fbcc1a8e700 1 -- 172.21.15.53:0/241160857 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcbc045ad0 con 0x7fbcbc070390 2024-01-21T19:58:23.835 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:23.842+0000 7fbcb3fff700 1 -- 172.21.15.53:0/241160857 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fbcac041020 con 0x7fbcbc070390 2024-01-21T19:58:23.835 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:23.844+0000 7fbcb3fff700 1 -- 172.21.15.53:0/241160857 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fbcac01add0 con 0x7fbcbc070390 2024-01-21T19:58:23.913 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:23.914+0000 7fbcc1a8e700 1 -- 172.21.15.53:0/241160857 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fbcbc101d70 con 0x7fbcbc070390 2024-01-21T19:58:23.913 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:23.915+0000 7fbcb3fff700 1 -- 172.21.15.53:0/241160857 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7fbcac0164e0 con 0x7fbcbc070390 2024-01-21T19:58:23.913 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:23.913 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: { 2024-01-21T19:58:23.913 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "fsid": "57f6722c-b897-11ee-95b0-87774f69a715", 2024-01-21T19:58:23.913 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "health": { 2024-01-21T19:58:23.913 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-21T19:58:23.913 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "checks": {}, 2024-01-21T19:58:23.913 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "mutes": [] 2024-01-21T19:58:23.913 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-01-21T19:58:23.913 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-21T19:58:23.914 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "quorum": [ 2024-01-21T19:58:23.914 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 0 2024-01-21T19:58:23.914 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: ], 2024-01-21T19:58:23.914 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-21T19:58:23.914 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "smithi053" 2024-01-21T19:58:23.914 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: ], 2024-01-21T19:58:23.914 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "quorum_age": 7, 2024-01-21T19:58:23.914 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "monmap": { 2024-01-21T19:58:23.914 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T19:58:23.914 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-21T19:58:23.914 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-21T19:58:23.914 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-01-21T19:58:23.914 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "osdmap": { 2024-01-21T19:58:23.914 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T19:58:23.914 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-21T19:58:23.915 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-21T19:58:23.915 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-21T19:58:23.915 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-21T19:58:23.915 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-21T19:58:23.915 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-21T19:58:23.915 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-01-21T19:58:23.915 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "pgmap": { 2024-01-21T19:58:23.915 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-21T19:58:23.915 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-21T19:58:23.915 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-21T19:58:23.915 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-21T19:58:23.915 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-21T19:58:23.915 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-21T19:58:23.915 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-21T19:58:23.916 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-21T19:58:23.916 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-01-21T19:58:23.916 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "fsmap": { 2024-01-21T19:58:23.916 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T19:58:23.916 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-21T19:58:23.916 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-21T19:58:23.916 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-01-21T19:58:23.916 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-21T19:58:23.916 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "available": false, 2024-01-21T19:58:23.916 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-21T19:58:23.916 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "modules": [ 2024-01-21T19:58:23.916 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "iostat", 2024-01-21T19:58:23.916 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "restful" 2024-01-21T19:58:23.916 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: ], 2024-01-21T19:58:23.916 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "services": {} 2024-01-21T19:58:23.917 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-01-21T19:58:23.917 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "servicemap": { 2024-01-21T19:58:23.917 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T19:58:23.917 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "modified": "2024-01-21T19:58:13.005835+0000", 2024-01-21T19:58:23.917 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "services": {} 2024-01-21T19:58:23.917 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-01-21T19:58:23.917 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-21T19:58:23.917 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: } 2024-01-21T19:58:23.917 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:23.915+0000 7fbcc1a8e700 1 -- 172.21.15.53:0/241160857 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbcbc070390 msgr2=0x7fbcbc10ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:23.917 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:23.915+0000 7fbcc1a8e700 1 --2- 172.21.15.53:0/241160857 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbcbc070390 0x7fbcbc10ccc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fbcac00b190 tx=0x7fbcac006ee0).stop 2024-01-21T19:58:23.917 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:23.915+0000 7fbcc1a8e700 1 -- 172.21.15.53:0/241160857 shutdown_connections 2024-01-21T19:58:23.917 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:23.915+0000 7fbcc1a8e700 1 --2- 172.21.15.53:0/241160857 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbcbc070390 0x7fbcbc10ccc0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:23.917 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:23.915+0000 7fbcc1a8e700 1 -- 172.21.15.53:0/241160857 >> 172.21.15.53:0/241160857 conn(0x7fbcbc0f63c0 msgr2=0x7fbcbc0f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:58:23.917 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:23.916+0000 7fbcc1a8e700 1 -- 172.21.15.53:0/241160857 shutdown_connections 2024-01-21T19:58:23.917 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:23.916+0000 7fbcc1a8e700 1 -- 172.21.15.53:0/241160857 wait complete. 2024-01-21T19:58:24.184 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:24 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:23.916172+0000 mon.smithi053 (mon.0) 22 : audit [DBG] from='client.? 172.21.15.53:0/241160857' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-21T19:58:24.184 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:24 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:58:24.003421+0000 mon.smithi053 (mon.0) 23 : cluster [DBG] mgrmap e3: smithi053.fgsmol(active, since 1.00541s) 2024-01-21T19:58:24.285 INFO:teuthology.orchestra.run.smithi053.stderr:mgr not available, waiting (3/15)... 2024-01-21T19:58:26.288 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/57f6722c-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpw7bt2for:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp987f45g0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-01-21T19:58:26.392 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:26 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:58:25.053200+0000 mon.smithi053 (mon.0) 24 : cluster [DBG] mgrmap e4: smithi053.fgsmol(active, since 2s) 2024-01-21T19:58:26.654 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.661+0000 7f050b65d700 1 Processor -- start 2024-01-21T19:58:26.654 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.662+0000 7f050b65d700 1 -- start start 2024-01-21T19:58:26.655 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.662+0000 7f050b65d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f050406fc10 0x7f05040fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:26.655 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.662+0000 7f050b65d700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05040fedd0 con 0x7f050406fc10 2024-01-21T19:58:26.655 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.662+0000 7f05093f9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f050406fc10 0x7f05040fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:26.655 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.662+0000 7f05093f9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f050406fc10 0x7f05040fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55076/0 (socket says 172.21.15.53:55076) 2024-01-21T19:58:26.655 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.662+0000 7f05093f9700 1 -- 172.21.15.53:0/3381770690 learned_addr learned my addr 172.21.15.53:0/3381770690 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:58:26.655 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.662+0000 7f05093f9700 1 -- 172.21.15.53:0/3381770690 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05040fef10 con 0x7f050406fc10 2024-01-21T19:58:26.658 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.662+0000 7f05093f9700 1 --2- 172.21.15.53:0/3381770690 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f050406fc10 0x7f05040fe8e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f050000b5f0 tx=0x7f0500006ee0).ready entity=mon.0 client_cookie=c1a0de40ca3e973f server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:26.658 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.662+0000 7f04f7fff700 1 -- 172.21.15.53:0/3381770690 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f050000d600 con 0x7f050406fc10 2024-01-21T19:58:26.658 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.663+0000 7f04f7fff700 1 -- 172.21.15.53:0/3381770690 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f050000dbd0 con 0x7f050406fc10 2024-01-21T19:58:26.658 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.663+0000 7f04f7fff700 1 -- 172.21.15.53:0/3381770690 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0500011c00 con 0x7f050406fc10 2024-01-21T19:58:26.658 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.663+0000 7f050b65d700 1 -- 172.21.15.53:0/3381770690 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f050406fc10 msgr2=0x7f05040fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:26.658 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.663+0000 7f050b65d700 1 --2- 172.21.15.53:0/3381770690 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f050406fc10 0x7f05040fe8e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f050000b5f0 tx=0x7f0500006ee0).stop 2024-01-21T19:58:26.658 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.663+0000 7f050b65d700 1 -- 172.21.15.53:0/3381770690 shutdown_connections 2024-01-21T19:58:26.658 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.663+0000 7f050b65d700 1 --2- 172.21.15.53:0/3381770690 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f050406fc10 0x7f05040fe8e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:26.658 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.663+0000 7f050b65d700 1 -- 172.21.15.53:0/3381770690 >> 172.21.15.53:0/3381770690 conn(0x7f05040f63a0 msgr2=0x7f05040f87c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:58:26.658 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.663+0000 7f050b65d700 1 -- 172.21.15.53:0/3381770690 shutdown_connections 2024-01-21T19:58:26.658 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.663+0000 7f050b65d700 1 -- 172.21.15.53:0/3381770690 wait complete. 2024-01-21T19:58:26.658 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.664+0000 7f050b65d700 1 Processor -- start 2024-01-21T19:58:26.659 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.664+0000 7f050b65d700 1 -- start start 2024-01-21T19:58:26.659 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.664+0000 7f050b65d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f050406fc10 0x7f050410a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:26.659 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.664+0000 7f050b65d700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f050410aee0 con 0x7f050406fc10 2024-01-21T19:58:26.659 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.665+0000 7f05093f9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f050406fc10 0x7f050410a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:26.659 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.665+0000 7f05093f9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f050406fc10 0x7f050410a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55082/0 (socket says 172.21.15.53:55082) 2024-01-21T19:58:26.659 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.665+0000 7f05093f9700 1 -- 172.21.15.53:0/1160401102 learned_addr learned my addr 172.21.15.53:0/1160401102 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:58:26.659 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.665+0000 7f05093f9700 1 -- 172.21.15.53:0/1160401102 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f050000b200 con 0x7f050406fc10 2024-01-21T19:58:26.659 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.665+0000 7f05093f9700 1 --2- 172.21.15.53:0/1160401102 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f050406fc10 0x7f050410a9f0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f050000b190 tx=0x7f0500006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:26.659 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.665+0000 7f04f5ffb700 1 -- 172.21.15.53:0/1160401102 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f050001b410 con 0x7f050406fc10 2024-01-21T19:58:26.659 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.665+0000 7f04f5ffb700 1 -- 172.21.15.53:0/1160401102 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f050001b9e0 con 0x7f050406fc10 2024-01-21T19:58:26.659 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.665+0000 7f04f5ffb700 1 -- 172.21.15.53:0/1160401102 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f050001a880 con 0x7f050406fc10 2024-01-21T19:58:26.659 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.665+0000 7f050b65d700 1 -- 172.21.15.53:0/1160401102 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f050410b0e0 con 0x7f050406fc10 2024-01-21T19:58:26.659 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.665+0000 7f050b65d700 1 -- 172.21.15.53:0/1160401102 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f050410b4f0 con 0x7f050406fc10 2024-01-21T19:58:26.660 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.666+0000 7f04f5ffb700 1 -- 172.21.15.53:0/1160401102 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7f050000f070 con 0x7f050406fc10 2024-01-21T19:58:26.660 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.667+0000 7f050b65d700 1 -- 172.21.15.53:0/1160401102 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0504045ad0 con 0x7f050406fc10 2024-01-21T19:58:26.660 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.667+0000 7f04f5ffb700 1 --2- 172.21.15.53:0/1160401102 >> [v2:172.21.15.53:6800/1431442886,v1:172.21.15.53:6801/1431442886] conn(0x7f04ec02fd50 0x7f04ec032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:26.660 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.667+0000 7f04f5ffb700 1 -- 172.21.15.53:0/1160401102 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f0500041090 con 0x7f050406fc10 2024-01-21T19:58:26.660 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.667+0000 7f0508bf8700 1 --2- 172.21.15.53:0/1160401102 >> [v2:172.21.15.53:6800/1431442886,v1:172.21.15.53:6801/1431442886] conn(0x7f04ec02fd50 0x7f04ec032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:26.661 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.667+0000 7f0508bf8700 1 --2- 172.21.15.53:0/1160401102 >> [v2:172.21.15.53:6800/1431442886,v1:172.21.15.53:6801/1431442886] conn(0x7f04ec02fd50 0x7f04ec032210 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f04f80096a0 tx=0x7f04f8006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:26.663 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.671+0000 7f04f5ffb700 1 -- 172.21.15.53:0/1160401102 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0500019950 con 0x7f050406fc10 2024-01-21T19:58:26.886 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.887+0000 7f050b65d700 1 -- 172.21.15.53:0/1160401102 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f05040ffb90 con 0x7f050406fc10 2024-01-21T19:58:26.887 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.887+0000 7f04f5ffb700 1 -- 172.21.15.53:0/1160401102 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f0500016070 con 0x7f050406fc10 2024-01-21T19:58:26.887 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.887 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: { 2024-01-21T19:58:26.887 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "fsid": "57f6722c-b897-11ee-95b0-87774f69a715", 2024-01-21T19:58:26.887 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "health": { 2024-01-21T19:58:26.887 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-21T19:58:26.887 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "checks": {}, 2024-01-21T19:58:26.887 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "mutes": [] 2024-01-21T19:58:26.887 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-01-21T19:58:26.887 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-21T19:58:26.887 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "quorum": [ 2024-01-21T19:58:26.887 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 0 2024-01-21T19:58:26.888 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: ], 2024-01-21T19:58:26.888 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-21T19:58:26.888 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "smithi053" 2024-01-21T19:58:26.888 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: ], 2024-01-21T19:58:26.888 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "quorum_age": 10, 2024-01-21T19:58:26.888 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "monmap": { 2024-01-21T19:58:26.888 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T19:58:26.888 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-21T19:58:26.888 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-21T19:58:26.888 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-01-21T19:58:26.888 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "osdmap": { 2024-01-21T19:58:26.888 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T19:58:26.888 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-21T19:58:26.889 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-21T19:58:26.889 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-21T19:58:26.889 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-21T19:58:26.889 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-21T19:58:26.889 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-21T19:58:26.889 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-01-21T19:58:26.889 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "pgmap": { 2024-01-21T19:58:26.889 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-21T19:58:26.889 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-21T19:58:26.889 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-21T19:58:26.889 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-21T19:58:26.889 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-21T19:58:26.889 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-21T19:58:26.889 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-21T19:58:26.889 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-21T19:58:26.890 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-01-21T19:58:26.890 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "fsmap": { 2024-01-21T19:58:26.890 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T19:58:26.890 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-21T19:58:26.890 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-21T19:58:26.890 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-01-21T19:58:26.890 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-21T19:58:26.890 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "available": true, 2024-01-21T19:58:26.890 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-21T19:58:26.890 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "modules": [ 2024-01-21T19:58:26.890 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "iostat", 2024-01-21T19:58:26.890 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "restful" 2024-01-21T19:58:26.890 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: ], 2024-01-21T19:58:26.890 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "services": {} 2024-01-21T19:58:26.890 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-01-21T19:58:26.890 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "servicemap": { 2024-01-21T19:58:26.891 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T19:58:26.891 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "modified": "2024-01-21T19:58:13.005835+0000", 2024-01-21T19:58:26.891 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "services": {} 2024-01-21T19:58:26.891 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-01-21T19:58:26.891 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-21T19:58:26.891 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: } 2024-01-21T19:58:26.891 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.889+0000 7f050b65d700 1 -- 172.21.15.53:0/1160401102 >> [v2:172.21.15.53:6800/1431442886,v1:172.21.15.53:6801/1431442886] conn(0x7f04ec02fd50 msgr2=0x7f04ec032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:26.891 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.889+0000 7f050b65d700 1 --2- 172.21.15.53:0/1160401102 >> [v2:172.21.15.53:6800/1431442886,v1:172.21.15.53:6801/1431442886] conn(0x7f04ec02fd50 0x7f04ec032210 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f04f80096a0 tx=0x7f04f8006b40).stop 2024-01-21T19:58:26.891 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.889+0000 7f050b65d700 1 -- 172.21.15.53:0/1160401102 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f050406fc10 msgr2=0x7f050410a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:26.891 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.889+0000 7f050b65d700 1 --2- 172.21.15.53:0/1160401102 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f050406fc10 0x7f050410a9f0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f050000b190 tx=0x7f0500006ee0).stop 2024-01-21T19:58:26.891 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.889+0000 7f050b65d700 1 -- 172.21.15.53:0/1160401102 shutdown_connections 2024-01-21T19:58:26.891 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.889+0000 7f050b65d700 1 --2- 172.21.15.53:0/1160401102 >> [v2:172.21.15.53:6800/1431442886,v1:172.21.15.53:6801/1431442886] conn(0x7f04ec02fd50 0x7f04ec032210 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:26.891 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.889+0000 7f050b65d700 1 --2- 172.21.15.53:0/1160401102 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f050406fc10 0x7f050410a9f0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:26.891 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.889+0000 7f050b65d700 1 -- 172.21.15.53:0/1160401102 >> 172.21.15.53:0/1160401102 conn(0x7f05040f63a0 msgr2=0x7f05040f7e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:58:26.891 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.889+0000 7f050b65d700 1 -- 172.21.15.53:0/1160401102 shutdown_connections 2024-01-21T19:58:26.891 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:26.889+0000 7f050b65d700 1 -- 172.21.15.53:0/1160401102 wait complete. 2024-01-21T19:58:27.244 INFO:teuthology.orchestra.run.smithi053.stderr:mgr is available 2024-01-21T19:58:27.244 INFO:teuthology.orchestra.run.smithi053.stderr:Enabling cephadm module... 2024-01-21T19:58:27.244 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/57f6722c-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpw7bt2for:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp987f45g0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2024-01-21T19:58:27.434 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:27 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:26.888696+0000 mon.smithi053 (mon.0) 25 : audit [DBG] from='client.? 172.21.15.53:0/1160401102' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-21T19:58:27.549 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:27.556+0000 7fb1b35b5700 1 Processor -- start 2024-01-21T19:58:27.549 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:27.556+0000 7fb1b35b5700 1 -- start start 2024-01-21T19:58:27.550 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:27.556+0000 7fb1b35b5700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1ac0703f0 0x7fb1ac0fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:27.550 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:27.556+0000 7fb1b35b5700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1ac0fee10 con 0x7fb1ac0703f0 2024-01-21T19:58:27.550 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:27.557+0000 7fb1b1351700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1ac0703f0 0x7fb1ac0fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:27.550 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:27.557+0000 7fb1b1351700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1ac0703f0 0x7fb1ac0fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55094/0 (socket says 172.21.15.53:55094) 2024-01-21T19:58:27.550 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:27.557+0000 7fb1b1351700 1 -- 172.21.15.53:0/2291586370 learned_addr learned my addr 172.21.15.53:0/2291586370 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:58:27.550 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:27.557+0000 7fb1b1351700 1 -- 172.21.15.53:0/2291586370 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1ac0fef50 con 0x7fb1ac0703f0 2024-01-21T19:58:27.550 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:27.557+0000 7fb1b1351700 1 --2- 172.21.15.53:0/2291586370 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1ac0703f0 0x7fb1ac0fe920 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fb19c00b5f0 tx=0x7fb19c006ee0).ready entity=mon.0 client_cookie=157f5f4f6497932f server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:27.550 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:27.557+0000 7fb19bfff700 1 -- 172.21.15.53:0/2291586370 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb19c00d600 con 0x7fb1ac0703f0 2024-01-21T19:58:27.550 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:27.557+0000 7fb19bfff700 1 -- 172.21.15.53:0/2291586370 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb19c00dbd0 con 0x7fb1ac0703f0 2024-01-21T19:58:27.553 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:27.557+0000 7fb19bfff700 1 -- 172.21.15.53:0/2291586370 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb19c011c00 con 0x7fb1ac0703f0 2024-01-21T19:58:27.553 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:27.558+0000 7fb1b35b5700 1 -- 172.21.15.53:0/2291586370 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1ac0703f0 msgr2=0x7fb1ac0fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:27.553 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:27.558+0000 7fb1b35b5700 1 --2- 172.21.15.53:0/2291586370 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1ac0703f0 0x7fb1ac0fe920 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fb19c00b5f0 tx=0x7fb19c006ee0).stop 2024-01-21T19:58:27.553 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:27.558+0000 7fb1b35b5700 1 -- 172.21.15.53:0/2291586370 shutdown_connections 2024-01-21T19:58:27.554 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:27.558+0000 7fb1b35b5700 1 --2- 172.21.15.53:0/2291586370 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1ac0703f0 0x7fb1ac0fe920 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:27.554 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:27.558+0000 7fb1b35b5700 1 -- 172.21.15.53:0/2291586370 >> 172.21.15.53:0/2291586370 conn(0x7fb1ac0f63c0 msgr2=0x7fb1ac0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:58:27.554 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:27.558+0000 7fb1b35b5700 1 -- 172.21.15.53:0/2291586370 shutdown_connections 2024-01-21T19:58:27.554 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:27.558+0000 7fb1b35b5700 1 -- 172.21.15.53:0/2291586370 wait complete. 2024-01-21T19:58:27.554 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:27.559+0000 7fb1b35b5700 1 Processor -- start 2024-01-21T19:58:27.554 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:27.559+0000 7fb1b35b5700 1 -- start start 2024-01-21T19:58:27.554 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:27.559+0000 7fb1b35b5700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1ac0703f0 0x7fb1ac0fc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:27.554 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:27.559+0000 7fb1b35b5700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1ac0fa820 con 0x7fb1ac0703f0 2024-01-21T19:58:27.554 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:27.559+0000 7fb1b1351700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1ac0703f0 0x7fb1ac0fc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:27.554 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:27.559+0000 7fb1b1351700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1ac0703f0 0x7fb1ac0fc3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55110/0 (socket says 172.21.15.53:55110) 2024-01-21T19:58:27.554 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:27.559+0000 7fb1b1351700 1 -- 172.21.15.53:0/1949839338 learned_addr learned my addr 172.21.15.53:0/1949839338 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:58:27.554 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:27.559+0000 7fb1b1351700 1 -- 172.21.15.53:0/1949839338 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb19c00b200 con 0x7fb1ac0703f0 2024-01-21T19:58:27.554 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:27.559+0000 7fb1b1351700 1 --2- 172.21.15.53:0/1949839338 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1ac0703f0 0x7fb1ac0fc3a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fb19c00b190 tx=0x7fb19c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:27.554 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:27.560+0000 7fb199ffb700 1 -- 172.21.15.53:0/1949839338 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb19c01b410 con 0x7fb1ac0703f0 2024-01-21T19:58:27.554 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:27.560+0000 7fb199ffb700 1 -- 172.21.15.53:0/1949839338 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb19c01b9e0 con 0x7fb1ac0703f0 2024-01-21T19:58:27.555 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:27.560+0000 7fb1b35b5700 1 -- 172.21.15.53:0/1949839338 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1ac0fa9c0 con 0x7fb1ac0703f0 2024-01-21T19:58:27.555 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:27.560+0000 7fb199ffb700 1 -- 172.21.15.53:0/1949839338 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb19c01a880 con 0x7fb1ac0703f0 2024-01-21T19:58:27.555 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:27.560+0000 7fb1b35b5700 1 -- 172.21.15.53:0/1949839338 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1ac0fadd0 con 0x7fb1ac0703f0 2024-01-21T19:58:27.555 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:27.560+0000 7fb199ffb700 1 -- 172.21.15.53:0/1949839338 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7fb19c00f070 con 0x7fb1ac0703f0 2024-01-21T19:58:27.555 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:27.561+0000 7fb199ffb700 1 --2- 172.21.15.53:0/1949839338 >> [v2:172.21.15.53:6800/1431442886,v1:172.21.15.53:6801/1431442886] conn(0x7fb19402fd50 0x7fb194032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:27.555 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:27.561+0000 7fb199ffb700 1 -- 172.21.15.53:0/1949839338 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fb19c041130 con 0x7fb1ac0703f0 2024-01-21T19:58:27.555 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:27.561+0000 7fb1b0b50700 1 --2- 172.21.15.53:0/1949839338 >> [v2:172.21.15.53:6800/1431442886,v1:172.21.15.53:6801/1431442886] conn(0x7fb19402fd50 0x7fb194032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:27.555 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:27.561+0000 7fb1b35b5700 1 -- 172.21.15.53:0/1949839338 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1ac059100 con 0x7fb1ac0703f0 2024-01-21T19:58:27.555 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:27.561+0000 7fb1b0b50700 1 --2- 172.21.15.53:0/1949839338 >> [v2:172.21.15.53:6800/1431442886,v1:172.21.15.53:6801/1431442886] conn(0x7fb19402fd50 0x7fb194032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fb1a80096a0 tx=0x7fb1a8006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:27.557 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:27.566+0000 7fb199ffb700 1 -- 172.21.15.53:0/1949839338 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb19c025460 con 0x7fb1ac0703f0 2024-01-21T19:58:27.754 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:27.762+0000 7fb1b35b5700 1 -- 172.21.15.53:0/1949839338 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fb1ac045ad0 con 0x7fb1ac0703f0 2024-01-21T19:58:28.412 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:28.421+0000 7fb199ffb700 1 -- 172.21.15.53:0/1949839338 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fb19c0257e0 con 0x7fb1ac0703f0 2024-01-21T19:58:28.413 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:28.422+0000 7fb199ffb700 1 -- 172.21.15.53:0/1949839338 <== mon.0 v2:172.21.15.53:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7fb19c022310 con 0x7fb1ac0703f0 2024-01-21T19:58:28.416 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:28.424+0000 7fb1b35b5700 1 -- 172.21.15.53:0/1949839338 >> [v2:172.21.15.53:6800/1431442886,v1:172.21.15.53:6801/1431442886] conn(0x7fb19402fd50 msgr2=0x7fb194032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:28.416 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:28.424+0000 7fb1b35b5700 1 --2- 172.21.15.53:0/1949839338 >> [v2:172.21.15.53:6800/1431442886,v1:172.21.15.53:6801/1431442886] conn(0x7fb19402fd50 0x7fb194032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fb1a80096a0 tx=0x7fb1a8006b40).stop 2024-01-21T19:58:28.416 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:28.424+0000 7fb1b35b5700 1 -- 172.21.15.53:0/1949839338 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1ac0703f0 msgr2=0x7fb1ac0fc3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:28.416 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:28.424+0000 7fb1b35b5700 1 --2- 172.21.15.53:0/1949839338 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1ac0703f0 0x7fb1ac0fc3a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fb19c00b190 tx=0x7fb19c006ee0).stop 2024-01-21T19:58:28.416 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:28.424+0000 7fb1b35b5700 1 -- 172.21.15.53:0/1949839338 shutdown_connections 2024-01-21T19:58:28.416 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:28.424+0000 7fb1b35b5700 1 --2- 172.21.15.53:0/1949839338 >> [v2:172.21.15.53:6800/1431442886,v1:172.21.15.53:6801/1431442886] conn(0x7fb19402fd50 0x7fb194032210 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:28.416 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:28.424+0000 7fb1b35b5700 1 --2- 172.21.15.53:0/1949839338 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1ac0703f0 0x7fb1ac0fc3a0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:28.416 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:28.424+0000 7fb1b35b5700 1 -- 172.21.15.53:0/1949839338 >> 172.21.15.53:0/1949839338 conn(0x7fb1ac0f63c0 msgr2=0x7fb1ac0f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:58:28.417 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:28.424+0000 7fb1b35b5700 1 -- 172.21.15.53:0/1949839338 shutdown_connections 2024-01-21T19:58:28.417 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:28.424+0000 7fb1b35b5700 1 -- 172.21.15.53:0/1949839338 wait complete. 2024-01-21T19:58:28.684 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:28 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:27.763689+0000 mon.smithi053 (mon.0) 26 : audit [INF] from='client.? 172.21.15.53:0/1949839338' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-01-21T19:58:28.818 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/57f6722c-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpw7bt2for:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp987f45g0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-01-21T19:58:29.193 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.200+0000 7f6457995700 1 Processor -- start 2024-01-21T19:58:29.193 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.200+0000 7f6457995700 1 -- start start 2024-01-21T19:58:29.193 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.200+0000 7f6457995700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6450070430 0x7f64500fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:29.194 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.201+0000 7f6457995700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64500fedd0 con 0x7f6450070430 2024-01-21T19:58:29.194 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.201+0000 7f6455731700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6450070430 0x7f64500fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:29.194 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.201+0000 7f6455731700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6450070430 0x7f64500fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55142/0 (socket says 172.21.15.53:55142) 2024-01-21T19:58:29.194 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.201+0000 7f6455731700 1 -- 172.21.15.53:0/56611438 learned_addr learned my addr 172.21.15.53:0/56611438 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:58:29.194 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.201+0000 7f6455731700 1 -- 172.21.15.53:0/56611438 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64500fef10 con 0x7f6450070430 2024-01-21T19:58:29.194 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.201+0000 7f6455731700 1 --2- 172.21.15.53:0/56611438 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6450070430 0x7f64500fe8e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f644c00b5f0 tx=0x7f644c006ee0).ready entity=mon.0 client_cookie=bc969cab15b7dab6 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:29.194 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.201+0000 7f643ffff700 1 -- 172.21.15.53:0/56611438 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f644c00d600 con 0x7f6450070430 2024-01-21T19:58:29.194 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.201+0000 7f643ffff700 1 -- 172.21.15.53:0/56611438 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f644c00dbd0 con 0x7f6450070430 2024-01-21T19:58:29.194 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.201+0000 7f643ffff700 1 -- 172.21.15.53:0/56611438 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f644c011c00 con 0x7f6450070430 2024-01-21T19:58:29.194 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.202+0000 7f6457995700 1 -- 172.21.15.53:0/56611438 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6450070430 msgr2=0x7f64500fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:29.194 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.202+0000 7f6457995700 1 --2- 172.21.15.53:0/56611438 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6450070430 0x7f64500fe8e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f644c00b5f0 tx=0x7f644c006ee0).stop 2024-01-21T19:58:29.194 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.202+0000 7f6457995700 1 -- 172.21.15.53:0/56611438 shutdown_connections 2024-01-21T19:58:29.194 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.202+0000 7f6457995700 1 --2- 172.21.15.53:0/56611438 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6450070430 0x7f64500fe8e0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:29.194 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.202+0000 7f6457995700 1 -- 172.21.15.53:0/56611438 >> 172.21.15.53:0/56611438 conn(0x7f64500f63c0 msgr2=0x7f64500f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:58:29.195 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.202+0000 7f6457995700 1 -- 172.21.15.53:0/56611438 shutdown_connections 2024-01-21T19:58:29.195 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.202+0000 7f6457995700 1 -- 172.21.15.53:0/56611438 wait complete. 2024-01-21T19:58:29.195 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.203+0000 7f6457995700 1 Processor -- start 2024-01-21T19:58:29.195 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.203+0000 7f6457995700 1 -- start start 2024-01-21T19:58:29.195 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.203+0000 7f6457995700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6450070430 0x7f645010aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:29.196 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.203+0000 7f6457995700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f645010af30 con 0x7f6450070430 2024-01-21T19:58:29.196 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.203+0000 7f6455731700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6450070430 0x7f645010aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:29.196 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.203+0000 7f6455731700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6450070430 0x7f645010aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55150/0 (socket says 172.21.15.53:55150) 2024-01-21T19:58:29.196 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.203+0000 7f6455731700 1 -- 172.21.15.53:0/2205562265 learned_addr learned my addr 172.21.15.53:0/2205562265 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:58:29.196 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.204+0000 7f6455731700 1 -- 172.21.15.53:0/2205562265 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f644c00b200 con 0x7f6450070430 2024-01-21T19:58:29.196 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.204+0000 7f6455731700 1 --2- 172.21.15.53:0/2205562265 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6450070430 0x7f645010aa40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f644c00e040 tx=0x7f644c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:29.196 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.204+0000 7f643dffb700 1 -- 172.21.15.53:0/2205562265 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f644c00d9f0 con 0x7f6450070430 2024-01-21T19:58:29.196 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.204+0000 7f6457995700 1 -- 172.21.15.53:0/2205562265 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f645010b130 con 0x7f6450070430 2024-01-21T19:58:29.196 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.204+0000 7f643dffb700 1 -- 172.21.15.53:0/2205562265 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f644c01a9f0 con 0x7f6450070430 2024-01-21T19:58:29.196 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.204+0000 7f643dffb700 1 -- 172.21.15.53:0/2205562265 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f644c01b8c0 con 0x7f6450070430 2024-01-21T19:58:29.196 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.204+0000 7f6457995700 1 -- 172.21.15.53:0/2205562265 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f645010b540 con 0x7f6450070430 2024-01-21T19:58:29.197 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.205+0000 7f643dffb700 1 -- 172.21.15.53:0/2205562265 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f644c00f070 con 0x7f6450070430 2024-01-21T19:58:29.197 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.205+0000 7f6457995700 1 -- 172.21.15.53:0/2205562265 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6450045ad0 con 0x7f6450070430 2024-01-21T19:58:29.197 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.205+0000 7f643dffb700 1 --2- 172.21.15.53:0/2205562265 >> [v2:172.21.15.53:6800/1431442886,v1:172.21.15.53:6801/1431442886] conn(0x7f643802fdb0 0x7f6438032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:29.197 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.205+0000 7f643dffb700 1 -- 172.21.15.53:0/2205562265 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f644c0411f0 con 0x7f6450070430 2024-01-21T19:58:29.198 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.206+0000 7f6454f30700 1 -- 172.21.15.53:0/2205562265 >> [v2:172.21.15.53:6800/1431442886,v1:172.21.15.53:6801/1431442886] conn(0x7f643802fdb0 msgr2=0x7f6438032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/1431442886 2024-01-21T19:58:29.198 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.206+0000 7f6454f30700 1 --2- 172.21.15.53:0/2205562265 >> [v2:172.21.15.53:6800/1431442886,v1:172.21.15.53:6801/1431442886] conn(0x7f643802fdb0 0x7f6438032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-21T19:58:29.200 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.210+0000 7f643dffb700 1 -- 172.21.15.53:0/2205562265 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f644c022d20 con 0x7f6450070430 2024-01-21T19:58:29.398 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.407+0000 7f6454f30700 1 -- 172.21.15.53:0/2205562265 >> [v2:172.21.15.53:6800/1431442886,v1:172.21.15.53:6801/1431442886] conn(0x7f643802fdb0 msgr2=0x7f6438032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/1431442886 2024-01-21T19:58:29.398 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.407+0000 7f6454f30700 1 --2- 172.21.15.53:0/2205562265 >> [v2:172.21.15.53:6800/1431442886,v1:172.21.15.53:6801/1431442886] conn(0x7f643802fdb0 0x7f6438032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-21T19:58:29.401 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.411+0000 7f6457995700 1 -- 172.21.15.53:0/2205562265 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f645010c5f0 con 0x7f6450070430 2024-01-21T19:58:29.402 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.411+0000 7f643dffb700 1 -- 172.21.15.53:0/2205562265 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f644c016070 con 0x7f6450070430 2024-01-21T19:58:29.402 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: { 2024-01-21T19:58:29.403 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 5, 2024-01-21T19:58:29.403 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "available": true, 2024-01-21T19:58:29.403 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "active_name": "smithi053.fgsmol", 2024-01-21T19:58:29.403 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_standby": 0 2024-01-21T19:58:29.403 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: } 2024-01-21T19:58:29.405 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.413+0000 7f6457995700 1 -- 172.21.15.53:0/2205562265 >> [v2:172.21.15.53:6800/1431442886,v1:172.21.15.53:6801/1431442886] conn(0x7f643802fdb0 msgr2=0x7f6438032270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-21T19:58:29.405 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.413+0000 7f6457995700 1 --2- 172.21.15.53:0/2205562265 >> [v2:172.21.15.53:6800/1431442886,v1:172.21.15.53:6801/1431442886] conn(0x7f643802fdb0 0x7f6438032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T19:58:29.405 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.413+0000 7f6457995700 1 -- 172.21.15.53:0/2205562265 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6450070430 msgr2=0x7f645010aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:29.405 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.413+0000 7f6457995700 1 --2- 172.21.15.53:0/2205562265 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6450070430 0x7f645010aa40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f644c00e040 tx=0x7f644c006ee0).stop 2024-01-21T19:58:29.405 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.413+0000 7f6457995700 1 -- 172.21.15.53:0/2205562265 shutdown_connections 2024-01-21T19:58:29.405 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.413+0000 7f6457995700 1 --2- 172.21.15.53:0/2205562265 >> [v2:172.21.15.53:6800/1431442886,v1:172.21.15.53:6801/1431442886] conn(0x7f643802fdb0 0x7f6438032270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T19:58:29.405 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.413+0000 7f6457995700 1 --2- 172.21.15.53:0/2205562265 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6450070430 0x7f645010aa40 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:29.405 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.413+0000 7f6457995700 1 -- 172.21.15.53:0/2205562265 >> 172.21.15.53:0/2205562265 conn(0x7f64500f63c0 msgr2=0x7f64500f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:58:29.405 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.414+0000 7f6457995700 1 -- 172.21.15.53:0/2205562265 shutdown_connections 2024-01-21T19:58:29.405 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:29.414+0000 7f6457995700 1 -- 172.21.15.53:0/2205562265 wait complete. 2024-01-21T19:58:29.684 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:29 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:28.422704+0000 mon.smithi053 (mon.0) 27 : audit [INF] from='client.? 172.21.15.53:0/1949839338' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-01-21T19:58:29.684 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:29 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:58:28.422752+0000 mon.smithi053 (mon.0) 28 : cluster [DBG] mgrmap e5: smithi053.fgsmol(active, since 5s) 2024-01-21T19:58:29.684 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:29 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:29.411703+0000 mon.smithi053 (mon.0) 29 : audit [DBG] from='client.? 172.21.15.53:0/2205562265' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-01-21T19:58:29.769 INFO:teuthology.orchestra.run.smithi053.stderr:Waiting for the mgr to restart... 2024-01-21T19:58:29.769 INFO:teuthology.orchestra.run.smithi053.stderr:Waiting for mgr epoch 5... 2024-01-21T19:58:29.769 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/57f6722c-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpw7bt2for:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp987f45g0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-01-21T19:58:30.113 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:30.120+0000 7fa1f6df4700 1 Processor -- start 2024-01-21T19:58:30.113 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:30.120+0000 7fa1f6df4700 1 -- start start 2024-01-21T19:58:30.113 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:30.120+0000 7fa1f6df4700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1f00703d0 0x7fa1f0100d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:30.113 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:30.120+0000 7fa1f6df4700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f00fc730 con 0x7fa1f00703d0 2024-01-21T19:58:30.113 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:30.120+0000 7fa1f4b90700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1f00703d0 0x7fa1f0100d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:30.113 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:30.120+0000 7fa1f4b90700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1f00703d0 0x7fa1f0100d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55166/0 (socket says 172.21.15.53:55166) 2024-01-21T19:58:30.113 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:30.120+0000 7fa1f4b90700 1 -- 172.21.15.53:0/1371932505 learned_addr learned my addr 172.21.15.53:0/1371932505 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:58:30.113 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:30.120+0000 7fa1f4b90700 1 -- 172.21.15.53:0/1371932505 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f0101230 con 0x7fa1f00703d0 2024-01-21T19:58:30.114 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:30.121+0000 7fa1f4b90700 1 --2- 172.21.15.53:0/1371932505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1f00703d0 0x7fa1f0100d40 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fa1e400b5f0 tx=0x7fa1e4006ee0).ready entity=mon.0 client_cookie=f16bcad071f51b8b server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:30.114 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:30.121+0000 7fa1ef7fe700 1 -- 172.21.15.53:0/1371932505 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1e400d600 con 0x7fa1f00703d0 2024-01-21T19:58:30.114 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:30.121+0000 7fa1ef7fe700 1 -- 172.21.15.53:0/1371932505 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa1e400dbd0 con 0x7fa1f00703d0 2024-01-21T19:58:30.114 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:30.121+0000 7fa1ef7fe700 1 -- 172.21.15.53:0/1371932505 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1e4011c00 con 0x7fa1f00703d0 2024-01-21T19:58:30.114 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:30.121+0000 7fa1f6df4700 1 -- 172.21.15.53:0/1371932505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1f00703d0 msgr2=0x7fa1f0100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:30.114 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:30.121+0000 7fa1f6df4700 1 --2- 172.21.15.53:0/1371932505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1f00703d0 0x7fa1f0100d40 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fa1e400b5f0 tx=0x7fa1e4006ee0).stop 2024-01-21T19:58:30.114 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:30.121+0000 7fa1f6df4700 1 -- 172.21.15.53:0/1371932505 shutdown_connections 2024-01-21T19:58:30.114 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:30.121+0000 7fa1f6df4700 1 --2- 172.21.15.53:0/1371932505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1f00703d0 0x7fa1f0100d40 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:30.114 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:30.121+0000 7fa1f6df4700 1 -- 172.21.15.53:0/1371932505 >> 172.21.15.53:0/1371932505 conn(0x7fa1f00f63c0 msgr2=0x7fa1f00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:58:30.114 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:30.121+0000 7fa1f6df4700 1 -- 172.21.15.53:0/1371932505 shutdown_connections 2024-01-21T19:58:30.114 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:30.121+0000 7fa1f6df4700 1 -- 172.21.15.53:0/1371932505 wait complete. 2024-01-21T19:58:30.114 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:30.122+0000 7fa1f6df4700 1 Processor -- start 2024-01-21T19:58:30.114 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:30.122+0000 7fa1f6df4700 1 -- start start 2024-01-21T19:58:30.117 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:30.123+0000 7fa1f6df4700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1f00703d0 0x7fa1f010cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:30.117 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:30.123+0000 7fa1f6df4700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f010d0a0 con 0x7fa1f00703d0 2024-01-21T19:58:30.117 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:30.123+0000 7fa1f4b90700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1f00703d0 0x7fa1f010cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:30.117 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:30.123+0000 7fa1f4b90700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1f00703d0 0x7fa1f010cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55182/0 (socket says 172.21.15.53:55182) 2024-01-21T19:58:30.117 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:30.123+0000 7fa1f4b90700 1 -- 172.21.15.53:0/3586117605 learned_addr learned my addr 172.21.15.53:0/3586117605 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:58:30.117 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:30.123+0000 7fa1f4b90700 1 -- 172.21.15.53:0/3586117605 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e400b200 con 0x7fa1f00703d0 2024-01-21T19:58:30.117 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:30.123+0000 7fa1f4b90700 1 --2- 172.21.15.53:0/3586117605 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1f00703d0 0x7fa1f010cbb0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fa1e400b190 tx=0x7fa1e4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:30.117 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:30.124+0000 7fa1ed7fa700 1 -- 172.21.15.53:0/3586117605 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1e401b410 con 0x7fa1f00703d0 2024-01-21T19:58:30.117 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:30.124+0000 7fa1ed7fa700 1 -- 172.21.15.53:0/3586117605 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa1e401b9e0 con 0x7fa1f00703d0 2024-01-21T19:58:30.117 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:30.124+0000 7fa1f6df4700 1 -- 172.21.15.53:0/3586117605 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1f010d2a0 con 0x7fa1f00703d0 2024-01-21T19:58:30.117 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:30.124+0000 7fa1ed7fa700 1 -- 172.21.15.53:0/3586117605 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1e401a880 con 0x7fa1f00703d0 2024-01-21T19:58:30.118 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:30.124+0000 7fa1f6df4700 1 -- 172.21.15.53:0/3586117605 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1f010d6b0 con 0x7fa1f00703d0 2024-01-21T19:58:30.118 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:30.125+0000 7fa1ed7fa700 1 -- 172.21.15.53:0/3586117605 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fa1e400f070 con 0x7fa1f00703d0 2024-01-21T19:58:30.118 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:30.125+0000 7fa1ed7fa700 1 --2- 172.21.15.53:0/3586117605 >> [v2:172.21.15.53:6800/1431442886,v1:172.21.15.53:6801/1431442886] conn(0x7fa1d402fdb0 0x7fa1d4032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:30.118 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:30.125+0000 7fa1ed7fa700 1 -- 172.21.15.53:0/3586117605 --> [v2:172.21.15.53:6800/1431442886,v1:172.21.15.53:6801/1431442886] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa1d4032930 con 0x7fa1d402fdb0 2024-01-21T19:58:30.118 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:30.125+0000 7fa1ed7fa700 1 -- 172.21.15.53:0/3586117605 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fa1e40410d0 con 0x7fa1f00703d0 2024-01-21T19:58:30.118 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:30.125+0000 7fa1effff700 1 -- 172.21.15.53:0/3586117605 >> [v2:172.21.15.53:6800/1431442886,v1:172.21.15.53:6801/1431442886] conn(0x7fa1d402fdb0 msgr2=0x7fa1d4032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/1431442886 2024-01-21T19:58:30.118 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:30.125+0000 7fa1effff700 1 --2- 172.21.15.53:0/3586117605 >> [v2:172.21.15.53:6800/1431442886,v1:172.21.15.53:6801/1431442886] conn(0x7fa1d402fdb0 0x7fa1d4032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-21T19:58:30.317 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:30.326+0000 7fa1effff700 1 -- 172.21.15.53:0/3586117605 >> [v2:172.21.15.53:6800/1431442886,v1:172.21.15.53:6801/1431442886] conn(0x7fa1d402fdb0 msgr2=0x7fa1d4032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/1431442886 2024-01-21T19:58:30.317 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:30.326+0000 7fa1effff700 1 --2- 172.21.15.53:0/3586117605 >> [v2:172.21.15.53:6800/1431442886,v1:172.21.15.53:6801/1431442886] conn(0x7fa1d402fdb0 0x7fa1d4032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-21T19:58:30.719 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:30.727+0000 7fa1effff700 1 -- 172.21.15.53:0/3586117605 >> [v2:172.21.15.53:6800/1431442886,v1:172.21.15.53:6801/1431442886] conn(0x7fa1d402fdb0 msgr2=0x7fa1d4032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/1431442886 2024-01-21T19:58:30.719 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:30.727+0000 7fa1effff700 1 --2- 172.21.15.53:0/3586117605 >> [v2:172.21.15.53:6800/1431442886,v1:172.21.15.53:6801/1431442886] conn(0x7fa1d402fdb0 0x7fa1d4032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-01-21T19:58:31.519 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:31.528+0000 7fa1effff700 1 -- 172.21.15.53:0/3586117605 >> [v2:172.21.15.53:6800/1431442886,v1:172.21.15.53:6801/1431442886] conn(0x7fa1d402fdb0 msgr2=0x7fa1d4032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/1431442886 2024-01-21T19:58:31.519 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:31.528+0000 7fa1effff700 1 --2- 172.21.15.53:0/3586117605 >> [v2:172.21.15.53:6800/1431442886,v1:172.21.15.53:6801/1431442886] conn(0x7fa1d402fdb0 0x7fa1d4032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-01-21T19:58:33.120 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:33.129+0000 7fa1effff700 1 -- 172.21.15.53:0/3586117605 >> [v2:172.21.15.53:6800/1431442886,v1:172.21.15.53:6801/1431442886] conn(0x7fa1d402fdb0 msgr2=0x7fa1d4032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/1431442886 2024-01-21T19:58:33.121 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:33.129+0000 7fa1effff700 1 --2- 172.21.15.53:0/3586117605 >> [v2:172.21.15.53:6800/1431442886,v1:172.21.15.53:6801/1431442886] conn(0x7fa1d402fdb0 0x7fa1d4032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-01-21T19:58:33.639 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:33.647+0000 7fa1ed7fa700 1 -- 172.21.15.53:0/3586117605 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mgrmap(e 6) v1 ==== 35970+0+0 (secure 0 0 0) 0x7fa1e401b570 con 0x7fa1f00703d0 2024-01-21T19:58:33.639 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:33.648+0000 7fa1ed7fa700 1 -- 172.21.15.53:0/3586117605 >> [v2:172.21.15.53:6800/1431442886,v1:172.21.15.53:6801/1431442886] conn(0x7fa1d402fdb0 msgr2=0x7fa1d4032270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-21T19:58:33.639 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:33.648+0000 7fa1ed7fa700 1 --2- 172.21.15.53:0/3586117605 >> [v2:172.21.15.53:6800/1431442886,v1:172.21.15.53:6801/1431442886] conn(0x7fa1d402fdb0 0x7fa1d4032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T19:58:34.640 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:34.649+0000 7fa1ed7fa700 1 -- 172.21.15.53:0/3586117605 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7fa1e4022d80 con 0x7fa1f00703d0 2024-01-21T19:58:34.640 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:34.649+0000 7fa1ed7fa700 1 --2- 172.21.15.53:0/3586117605 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7fa1d4033880 0x7fa1d4035c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:34.641 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:34.649+0000 7fa1ed7fa700 1 -- 172.21.15.53:0/3586117605 --> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa1d4032930 con 0x7fa1d4033880 2024-01-21T19:58:34.641 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:34.650+0000 7fa1effff700 1 --2- 172.21.15.53:0/3586117605 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7fa1d4033880 0x7fa1d4035c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:34.641 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:34.650+0000 7fa1effff700 1 --2- 172.21.15.53:0/3586117605 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7fa1d4033880 0x7fa1d4035c70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fa1dc006fd0 tx=0x7fa1dc006d70).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:34.641 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:34.651+0000 7fa1ed7fa700 1 -- 172.21.15.53:0/3586117605 <== mgr.14116 v2:172.21.15.53:6800/55197364 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7fa1d4032930 con 0x7fa1d4033880 2024-01-21T19:58:34.648 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:34.657+0000 7fa1f6df4700 1 -- 172.21.15.53:0/3586117605 --> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fa1f0059100 con 0x7fa1d4033880 2024-01-21T19:58:34.649 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:34.658+0000 7fa1ed7fa700 1 -- 172.21.15.53:0/3586117605 <== mgr.14116 v2:172.21.15.53:6800/55197364 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fa1f0059100 con 0x7fa1d4033880 2024-01-21T19:58:34.649 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: { 2024-01-21T19:58:34.649 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-01-21T19:58:34.649 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "initialized": true 2024-01-21T19:58:34.650 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: } 2024-01-21T19:58:34.650 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:34.658+0000 7fa1f6df4700 1 -- 172.21.15.53:0/3586117605 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7fa1d4033880 msgr2=0x7fa1d4035c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:34.651 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:34.658+0000 7fa1f6df4700 1 --2- 172.21.15.53:0/3586117605 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7fa1d4033880 0x7fa1d4035c70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fa1dc006fd0 tx=0x7fa1dc006d70).stop 2024-01-21T19:58:34.651 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:34.658+0000 7fa1f6df4700 1 -- 172.21.15.53:0/3586117605 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1f00703d0 msgr2=0x7fa1f010cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:34.651 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:34.658+0000 7fa1f6df4700 1 --2- 172.21.15.53:0/3586117605 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1f00703d0 0x7fa1f010cbb0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fa1e400b190 tx=0x7fa1e4006ee0).stop 2024-01-21T19:58:34.651 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:34.658+0000 7fa1f6df4700 1 -- 172.21.15.53:0/3586117605 shutdown_connections 2024-01-21T19:58:34.651 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:34.658+0000 7fa1f6df4700 1 --2- 172.21.15.53:0/3586117605 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1f00703d0 0x7fa1f010cbb0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:34.651 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:34.658+0000 7fa1f6df4700 1 --2- 172.21.15.53:0/3586117605 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7fa1d4033880 0x7fa1d4035c70 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:34.651 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:34.658+0000 7fa1f6df4700 1 --2- 172.21.15.53:0/3586117605 >> [v2:172.21.15.53:6800/1431442886,v1:172.21.15.53:6801/1431442886] conn(0x7fa1d402fdb0 0x7fa1d4032270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T19:58:34.651 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:34.658+0000 7fa1f6df4700 1 -- 172.21.15.53:0/3586117605 >> 172.21.15.53:0/3586117605 conn(0x7fa1f00f63c0 msgr2=0x7fa1f00f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:58:34.651 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:34.658+0000 7fa1f6df4700 1 -- 172.21.15.53:0/3586117605 shutdown_connections 2024-01-21T19:58:34.651 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:34.658+0000 7fa1f6df4700 1 -- 172.21.15.53:0/3586117605 wait complete. 2024-01-21T19:58:34.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:34 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:58:33.596258+0000 mon.smithi053 (mon.0) 30 : cluster [INF] Active manager daemon smithi053.fgsmol restarted 2024-01-21T19:58:34.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:34 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:58:33.596666+0000 mon.smithi053 (mon.0) 31 : cluster [INF] Activating manager daemon smithi053.fgsmol 2024-01-21T19:58:34.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:34 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:58:33.598250+0000 mon.smithi053 (mon.0) 32 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-01-21T19:58:34.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:34 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:58:33.649244+0000 mon.smithi053 (mon.0) 33 : cluster [DBG] mgrmap e6: smithi053.fgsmol(active, starting, since 0.0526796s) 2024-01-21T19:58:34.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:34 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:33.650409+0000 mon.smithi053 (mon.0) 34 : audit [DBG] from='mgr.14116 172.21.15.53:0/1503870519' entity='mgr.smithi053.fgsmol' cmd=[{"prefix": "mon metadata", "id": "smithi053"}]: dispatch 2024-01-21T19:58:34.935 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:34 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:33.650947+0000 mon.smithi053 (mon.0) 35 : audit [DBG] from='mgr.14116 172.21.15.53:0/1503870519' entity='mgr.smithi053.fgsmol' cmd=[{"prefix": "mgr metadata", "who": "smithi053.fgsmol", "id": "smithi053.fgsmol"}]: dispatch 2024-01-21T19:58:34.935 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:34 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:33.651624+0000 mon.smithi053 (mon.0) 36 : audit [DBG] from='mgr.14116 172.21.15.53:0/1503870519' entity='mgr.smithi053.fgsmol' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-21T19:58:34.935 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:34 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:33.651820+0000 mon.smithi053 (mon.0) 37 : audit [DBG] from='mgr.14116 172.21.15.53:0/1503870519' entity='mgr.smithi053.fgsmol' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-21T19:58:34.935 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:34 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:33.652012+0000 mon.smithi053 (mon.0) 38 : audit [DBG] from='mgr.14116 172.21.15.53:0/1503870519' entity='mgr.smithi053.fgsmol' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-21T19:58:34.935 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:34 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:58:33.665803+0000 mon.smithi053 (mon.0) 39 : cluster [INF] Manager daemon smithi053.fgsmol is now available 2024-01-21T19:58:34.935 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:34 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:33.680278+0000 mon.smithi053 (mon.0) 40 : audit [INF] from='mgr.14116 172.21.15.53:0/1503870519' entity='mgr.smithi053.fgsmol' 2024-01-21T19:58:34.935 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:34 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:33.683020+0000 mon.smithi053 (mon.0) 41 : audit [INF] from='mgr.14116 172.21.15.53:0/1503870519' entity='mgr.smithi053.fgsmol' 2024-01-21T19:58:34.935 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:34 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:33.685452+0000 mon.smithi053 (mon.0) 42 : audit [INF] from='mgr.14116 172.21.15.53:0/1503870519' entity='mgr.smithi053.fgsmol' 2024-01-21T19:58:34.935 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:34 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:33.780122+0000 mon.smithi053 (mon.0) 43 : audit [INF] from='mgr.14116 172.21.15.53:0/1503870519' entity='mgr.smithi053.fgsmol' 2024-01-21T19:58:34.935 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:34 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:33.780872+0000 mon.smithi053 (mon.0) 44 : audit [DBG] from='mgr.14116 172.21.15.53:0/1503870519' entity='mgr.smithi053.fgsmol' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T19:58:34.935 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:34 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:33.782496+0000 mon.smithi053 (mon.0) 45 : audit [DBG] from='mgr.14116 172.21.15.53:0/1503870519' entity='mgr.smithi053.fgsmol' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T19:58:34.935 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:34 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:33.784140+0000 mon.smithi053 (mon.0) 46 : audit [DBG] from='mgr.14116 172.21.15.53:0/1503870519' entity='mgr.smithi053.fgsmol' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T19:58:34.935 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:34 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:33.794375+0000 mon.smithi053 (mon.0) 47 : audit [INF] from='mgr.14116 172.21.15.53:0/1503870519' entity='mgr.smithi053.fgsmol' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fgsmol/mirror_snapshot_schedule"}]: dispatch 2024-01-21T19:58:34.935 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:34 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:33.795513+0000 mon.smithi053 (mon.0) 48 : audit [INF] from='mgr.14116 172.21.15.53:0/1503870519' entity='mgr.smithi053.fgsmol' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fgsmol/trash_purge_schedule"}]: dispatch 2024-01-21T19:58:35.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:35 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:34.651220+0000 mgr.smithi053.fgsmol (mgr.14116) 1 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-01-21T19:58:35.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:35 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:58:34.652107+0000 mon.smithi053 (mon.0) 49 : cluster [DBG] mgrmap e7: smithi053.fgsmol(active, since 1.05554s) 2024-01-21T19:58:35.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:35 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:34.658221+0000 mgr.smithi053.fgsmol (mgr.14116) 2 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-01-21T19:58:36.327 INFO:teuthology.orchestra.run.smithi053.stderr:mgr epoch 5 is available 2024-01-21T19:58:36.327 INFO:teuthology.orchestra.run.smithi053.stderr:Setting orchestrator backend to cephadm... 2024-01-21T19:58:36.327 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/57f6722c-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpw7bt2for:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp987f45g0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2024-01-21T19:58:36.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:36 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:58:35.654126+0000 mon.smithi053 (mon.0) 50 : cluster [DBG] mgrmap e8: smithi053.fgsmol(active, since 2s) 2024-01-21T19:58:37.766 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.775+0000 7f13489b3700 1 Processor -- start 2024-01-21T19:58:37.766 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.775+0000 7f13489b3700 1 -- start start 2024-01-21T19:58:37.766 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.775+0000 7f13489b3700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1344070390 0x7f1344100d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:37.767 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.775+0000 7f13489b3700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13440fc770 con 0x7f1344070390 2024-01-21T19:58:37.768 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.775+0000 7f134259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1344070390 0x7f1344100d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:37.768 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.775+0000 7f134259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1344070390 0x7f1344100d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49304/0 (socket says 172.21.15.53:49304) 2024-01-21T19:58:37.768 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.775+0000 7f134259c700 1 -- 172.21.15.53:0/1879306526 learned_addr learned my addr 172.21.15.53:0/1879306526 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:58:37.768 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.775+0000 7f134259c700 1 -- 172.21.15.53:0/1879306526 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1344101270 con 0x7f1344070390 2024-01-21T19:58:37.768 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.776+0000 7f134259c700 1 --2- 172.21.15.53:0/1879306526 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1344070390 0x7f1344100d80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f1334009960 tx=0x7f13340092d0).ready entity=mon.0 client_cookie=2a8120bf5d1624c1 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:37.768 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.776+0000 7f1341d9b700 1 -- 172.21.15.53:0/1879306526 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f133400b9c0 con 0x7f1344070390 2024-01-21T19:58:37.768 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.776+0000 7f1341d9b700 1 -- 172.21.15.53:0/1879306526 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1334011400 con 0x7f1344070390 2024-01-21T19:58:37.768 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.776+0000 7f1341d9b700 1 -- 172.21.15.53:0/1879306526 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13340103a0 con 0x7f1344070390 2024-01-21T19:58:37.768 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.777+0000 7f13489b3700 1 -- 172.21.15.53:0/1879306526 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1344070390 msgr2=0x7f1344100d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:37.769 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.777+0000 7f13489b3700 1 --2- 172.21.15.53:0/1879306526 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1344070390 0x7f1344100d80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f1334009960 tx=0x7f13340092d0).stop 2024-01-21T19:58:37.771 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.777+0000 7f13489b3700 1 -- 172.21.15.53:0/1879306526 shutdown_connections 2024-01-21T19:58:37.771 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.777+0000 7f13489b3700 1 --2- 172.21.15.53:0/1879306526 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1344070390 0x7f1344100d80 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:37.771 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.777+0000 7f13489b3700 1 -- 172.21.15.53:0/1879306526 >> 172.21.15.53:0/1879306526 conn(0x7f13440f63c0 msgr2=0x7f13440f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:58:37.771 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.777+0000 7f13489b3700 1 -- 172.21.15.53:0/1879306526 shutdown_connections 2024-01-21T19:58:37.772 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.777+0000 7f13489b3700 1 -- 172.21.15.53:0/1879306526 wait complete. 2024-01-21T19:58:37.772 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.777+0000 7f13489b3700 1 Processor -- start 2024-01-21T19:58:37.772 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.778+0000 7f13489b3700 1 -- start start 2024-01-21T19:58:37.772 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.778+0000 7f13489b3700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1344070390 0x7f134410cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:37.772 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.778+0000 7f13489b3700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f134410d0f0 con 0x7f1344070390 2024-01-21T19:58:37.772 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.778+0000 7f134259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1344070390 0x7f134410cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:37.772 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.778+0000 7f134259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1344070390 0x7f134410cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49314/0 (socket says 172.21.15.53:49314) 2024-01-21T19:58:37.772 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.778+0000 7f134259c700 1 -- 172.21.15.53:0/1176448798 learned_addr learned my addr 172.21.15.53:0/1176448798 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:58:37.772 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.778+0000 7f134259c700 1 -- 172.21.15.53:0/1176448798 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1334009570 con 0x7f1344070390 2024-01-21T19:58:37.772 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.778+0000 7f134259c700 1 --2- 172.21.15.53:0/1176448798 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1344070390 0x7f134410cc00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f1334011c70 tx=0x7f13340092d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:37.772 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.778+0000 7f133b7fe700 1 -- 172.21.15.53:0/1176448798 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1334010870 con 0x7f1344070390 2024-01-21T19:58:37.772 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.778+0000 7f133b7fe700 1 -- 172.21.15.53:0/1176448798 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1334010e40 con 0x7f1344070390 2024-01-21T19:58:37.772 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.778+0000 7f133b7fe700 1 -- 172.21.15.53:0/1176448798 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f133401ae30 con 0x7f1344070390 2024-01-21T19:58:37.772 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.778+0000 7f13489b3700 1 -- 172.21.15.53:0/1176448798 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f134410d2f0 con 0x7f1344070390 2024-01-21T19:58:37.773 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.779+0000 7f13489b3700 1 -- 172.21.15.53:0/1176448798 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f134410d700 con 0x7f1344070390 2024-01-21T19:58:37.773 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.779+0000 7f133b7fe700 1 -- 172.21.15.53:0/1176448798 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f133402a070 con 0x7f1344070390 2024-01-21T19:58:37.773 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.780+0000 7f13489b3700 1 -- 172.21.15.53:0/1176448798 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1344045ad0 con 0x7f1344070390 2024-01-21T19:58:37.773 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.780+0000 7f133b7fe700 1 --2- 172.21.15.53:0/1176448798 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7f132402fd00 0x7f13240321c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:37.773 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.780+0000 7f133b7fe700 1 -- 172.21.15.53:0/1176448798 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f1334041ab0 con 0x7f1344070390 2024-01-21T19:58:37.773 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.780+0000 7f1339d9b700 1 --2- 172.21.15.53:0/1176448798 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7f132402fd00 0x7f13240321c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:37.773 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.780+0000 7f1339d9b700 1 --2- 172.21.15.53:0/1176448798 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7f132402fd00 0x7f13240321c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f132c0096a0 tx=0x7f132c006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:37.775 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.784+0000 7f133b7fe700 1 -- 172.21.15.53:0/1176448798 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1334018d60 con 0x7f1344070390 2024-01-21T19:58:37.965 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.974+0000 7f13489b3700 1 -- 172.21.15.53:0/1176448798 --> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f134410df70 con 0x7f132402fd00 2024-01-21T19:58:37.969 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.979+0000 7f133b7fe700 1 -- 172.21.15.53:0/1176448798 <== mgr.14116 v2:172.21.15.53:6800/55197364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f134410df70 con 0x7f132402fd00 2024-01-21T19:58:37.972 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.981+0000 7f13489b3700 1 -- 172.21.15.53:0/1176448798 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7f132402fd00 msgr2=0x7f13240321c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:37.973 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.981+0000 7f13489b3700 1 --2- 172.21.15.53:0/1176448798 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7f132402fd00 0x7f13240321c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f132c0096a0 tx=0x7f132c006b40).stop 2024-01-21T19:58:37.973 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.981+0000 7f13489b3700 1 -- 172.21.15.53:0/1176448798 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1344070390 msgr2=0x7f134410cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:37.973 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.981+0000 7f13489b3700 1 --2- 172.21.15.53:0/1176448798 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1344070390 0x7f134410cc00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f1334011c70 tx=0x7f13340092d0).stop 2024-01-21T19:58:37.973 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.981+0000 7f13489b3700 1 -- 172.21.15.53:0/1176448798 shutdown_connections 2024-01-21T19:58:37.973 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.981+0000 7f13489b3700 1 --2- 172.21.15.53:0/1176448798 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7f132402fd00 0x7f13240321c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:37.973 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.981+0000 7f13489b3700 1 --2- 172.21.15.53:0/1176448798 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1344070390 0x7f134410cc00 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:37.973 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.981+0000 7f13489b3700 1 -- 172.21.15.53:0/1176448798 >> 172.21.15.53:0/1176448798 conn(0x7f13440f63c0 msgr2=0x7f13440f7ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:58:37.973 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.982+0000 7f13489b3700 1 -- 172.21.15.53:0/1176448798 shutdown_connections 2024-01-21T19:58:37.973 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:37.982+0000 7f13489b3700 1 -- 172.21.15.53:0/1176448798 wait complete. 2024-01-21T19:58:39.410 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:38 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:37.975347+0000 mgr.smithi053.fgsmol (mgr.14116) 3 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T19:58:39.410 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:38 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:37.979027+0000 mon.smithi053 (mon.0) 51 : audit [INF] from='mgr.14116 172.21.15.53:0/1503870519' entity='mgr.smithi053.fgsmol' 2024-01-21T19:58:39.410 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:38 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:37.983191+0000 mon.smithi053 (mon.0) 52 : audit [DBG] from='mgr.14116 172.21.15.53:0/1503870519' entity='mgr.smithi053.fgsmol' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T19:58:40.768 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/57f6722c-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpw7bt2for:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp987f45g0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2024-01-21T19:58:42.068 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:41.950+0000 7faaf71ae700 1 Processor -- start 2024-01-21T19:58:42.069 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:41.950+0000 7faaf71ae700 1 -- start start 2024-01-21T19:58:42.069 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:41.950+0000 7faaf71ae700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faaf0070370 0x7faaf00fe9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:42.069 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:41.950+0000 7faaf71ae700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf0070790 con 0x7faaf0070370 2024-01-21T19:58:42.069 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:41.950+0000 7faaf4f4a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faaf0070370 0x7faaf00fe9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:42.069 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:41.950+0000 7faaf4f4a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faaf0070370 0x7faaf00fe9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49316/0 (socket says 172.21.15.53:49316) 2024-01-21T19:58:42.069 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:41.950+0000 7faaf4f4a700 1 -- 172.21.15.53:0/3355160470 learned_addr learned my addr 172.21.15.53:0/3355160470 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:58:42.069 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:41.950+0000 7faaf4f4a700 1 -- 172.21.15.53:0/3355160470 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaf00fee90 con 0x7faaf0070370 2024-01-21T19:58:42.069 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:41.950+0000 7faaf4f4a700 1 --2- 172.21.15.53:0/3355160470 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faaf0070370 0x7faaf00fe9a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7faae400b5f0 tx=0x7faae4006ee0).ready entity=mon.0 client_cookie=f954d6617c3dae92 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:42.069 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:41.951+0000 7faaef7fe700 1 -- 172.21.15.53:0/3355160470 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faae400d600 con 0x7faaf0070370 2024-01-21T19:58:42.069 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:41.951+0000 7faaef7fe700 1 -- 172.21.15.53:0/3355160470 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7faae400dbd0 con 0x7faaf0070370 2024-01-21T19:58:42.069 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:41.951+0000 7faaef7fe700 1 -- 172.21.15.53:0/3355160470 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faae4011c00 con 0x7faaf0070370 2024-01-21T19:58:42.069 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:41.951+0000 7faaf71ae700 1 -- 172.21.15.53:0/3355160470 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faaf0070370 msgr2=0x7faaf00fe9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:42.069 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:41.951+0000 7faaf71ae700 1 --2- 172.21.15.53:0/3355160470 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faaf0070370 0x7faaf00fe9a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7faae400b5f0 tx=0x7faae4006ee0).stop 2024-01-21T19:58:42.070 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:41.951+0000 7faaf71ae700 1 -- 172.21.15.53:0/3355160470 shutdown_connections 2024-01-21T19:58:42.070 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:41.951+0000 7faaf71ae700 1 --2- 172.21.15.53:0/3355160470 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faaf0070370 0x7faaf00fe9a0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:42.070 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:41.951+0000 7faaf71ae700 1 -- 172.21.15.53:0/3355160470 >> 172.21.15.53:0/3355160470 conn(0x7faaf00f63c0 msgr2=0x7faaf00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:58:42.070 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:41.951+0000 7faaf71ae700 1 -- 172.21.15.53:0/3355160470 shutdown_connections 2024-01-21T19:58:42.070 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:41.951+0000 7faaf71ae700 1 -- 172.21.15.53:0/3355160470 wait complete. 2024-01-21T19:58:42.070 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:41.952+0000 7faaf71ae700 1 Processor -- start 2024-01-21T19:58:42.070 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:41.952+0000 7faaf71ae700 1 -- start start 2024-01-21T19:58:42.070 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:41.952+0000 7faaf71ae700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faaf0070370 0x7faaf010aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:42.070 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:41.952+0000 7faaf71ae700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf010aef0 con 0x7faaf0070370 2024-01-21T19:58:42.070 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:41.952+0000 7faaf4f4a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faaf0070370 0x7faaf010aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:42.070 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:41.952+0000 7faaf4f4a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faaf0070370 0x7faaf010aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49330/0 (socket says 172.21.15.53:49330) 2024-01-21T19:58:42.070 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:41.952+0000 7faaf4f4a700 1 -- 172.21.15.53:0/1848347687 learned_addr learned my addr 172.21.15.53:0/1848347687 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:58:42.070 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:41.953+0000 7faaf4f4a700 1 -- 172.21.15.53:0/1848347687 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae400b200 con 0x7faaf0070370 2024-01-21T19:58:42.070 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:41.953+0000 7faaf4f4a700 1 --2- 172.21.15.53:0/1848347687 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faaf0070370 0x7faaf010aa00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7faae400e040 tx=0x7faae4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:42.070 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:41.953+0000 7faaed7fa700 1 -- 172.21.15.53:0/1848347687 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faae400d9f0 con 0x7faaf0070370 2024-01-21T19:58:42.071 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:41.953+0000 7faaed7fa700 1 -- 172.21.15.53:0/1848347687 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7faae401a9f0 con 0x7faaf0070370 2024-01-21T19:58:42.071 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:41.953+0000 7faaf71ae700 1 -- 172.21.15.53:0/1848347687 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaf010b0f0 con 0x7faaf0070370 2024-01-21T19:58:42.071 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:41.953+0000 7faaed7fa700 1 -- 172.21.15.53:0/1848347687 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faae401b890 con 0x7faaf0070370 2024-01-21T19:58:42.071 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:41.953+0000 7faaf71ae700 1 -- 172.21.15.53:0/1848347687 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaf010b500 con 0x7faaf0070370 2024-01-21T19:58:42.071 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:41.954+0000 7faaed7fa700 1 -- 172.21.15.53:0/1848347687 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7faae400f070 con 0x7faaf0070370 2024-01-21T19:58:42.071 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:41.954+0000 7faaed7fa700 1 --2- 172.21.15.53:0/1848347687 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7faad402fdb0 0x7faad4032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:42.071 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:41.954+0000 7faaed7fa700 1 -- 172.21.15.53:0/1848347687 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7faae40412e0 con 0x7faaf0070370 2024-01-21T19:58:42.071 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:41.954+0000 7faaf71ae700 1 -- 172.21.15.53:0/1848347687 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaf0045ad0 con 0x7faaf0070370 2024-01-21T19:58:42.071 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:41.954+0000 7faaeffff700 1 --2- 172.21.15.53:0/1848347687 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7faad402fdb0 0x7faad4032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:42.071 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:41.954+0000 7faaeffff700 1 --2- 172.21.15.53:0/1848347687 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7faad402fdb0 0x7faad4032270 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7faadc0096a0 tx=0x7faadc006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:42.071 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:41.959+0000 7faaed7fa700 1 -- 172.21.15.53:0/1848347687 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faae4016080 con 0x7faaf0070370 2024-01-21T19:58:42.112 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:42.120+0000 7faaf71ae700 1 -- 172.21.15.53:0/1848347687 --> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7faaf01048a0 con 0x7faad402fdb0 2024-01-21T19:58:42.112 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:42.120+0000 7faaed7fa700 1 -- 172.21.15.53:0/1848347687 <== mgr.14116 v2:172.21.15.53:6800/55197364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7faaf01048a0 con 0x7faad402fdb0 2024-01-21T19:58:42.112 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: value unchanged 2024-01-21T19:58:42.115 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:42.122+0000 7faaf71ae700 1 -- 172.21.15.53:0/1848347687 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7faad402fdb0 msgr2=0x7faad4032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:42.115 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:42.122+0000 7faaf71ae700 1 --2- 172.21.15.53:0/1848347687 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7faad402fdb0 0x7faad4032270 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7faadc0096a0 tx=0x7faadc006b40).stop 2024-01-21T19:58:42.115 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:42.122+0000 7faaf71ae700 1 -- 172.21.15.53:0/1848347687 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faaf0070370 msgr2=0x7faaf010aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:42.115 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:42.122+0000 7faaf71ae700 1 --2- 172.21.15.53:0/1848347687 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faaf0070370 0x7faaf010aa00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7faae400e040 tx=0x7faae4006ee0).stop 2024-01-21T19:58:42.115 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:42.122+0000 7faaf71ae700 1 -- 172.21.15.53:0/1848347687 shutdown_connections 2024-01-21T19:58:42.115 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:42.122+0000 7faaf71ae700 1 --2- 172.21.15.53:0/1848347687 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7faad402fdb0 0x7faad4032270 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:42.115 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:42.122+0000 7faaf71ae700 1 --2- 172.21.15.53:0/1848347687 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faaf0070370 0x7faaf010aa00 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:42.115 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:42.122+0000 7faaf71ae700 1 -- 172.21.15.53:0/1848347687 >> 172.21.15.53:0/1848347687 conn(0x7faaf00f63c0 msgr2=0x7faaf00f7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:58:42.115 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:42.123+0000 7faaf71ae700 1 -- 172.21.15.53:0/1848347687 shutdown_connections 2024-01-21T19:58:42.116 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:42.123+0000 7faaf71ae700 1 -- 172.21.15.53:0/1848347687 wait complete. 2024-01-21T19:58:42.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:42 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:42.122012+0000 mgr.smithi053.fgsmol (mgr.14116) 4 : audit [DBG] from='client.14128 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T19:58:43.806 INFO:teuthology.orchestra.run.smithi053.stderr:Generating ssh key... 2024-01-21T19:58:43.806 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/57f6722c-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpw7bt2for:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp987f45g0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2024-01-21T19:58:44.330 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.335+0000 7f915ea44700 1 Processor -- start 2024-01-21T19:58:44.330 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.335+0000 7f915ea44700 1 -- start start 2024-01-21T19:58:44.330 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.335+0000 7f915ea44700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91580724d0 0x7f9158070a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:44.331 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.335+0000 7f915ea44700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91580728f0 con 0x7f91580724d0 2024-01-21T19:58:44.331 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.335+0000 7f9157fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91580724d0 0x7f9158070a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:44.331 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.335+0000 7f9157fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91580724d0 0x7f9158070a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49342/0 (socket says 172.21.15.53:49342) 2024-01-21T19:58:44.331 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.335+0000 7f9157fff700 1 -- 172.21.15.53:0/165004094 learned_addr learned my addr 172.21.15.53:0/165004094 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:58:44.331 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.335+0000 7f9157fff700 1 -- 172.21.15.53:0/165004094 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9158070f60 con 0x7f91580724d0 2024-01-21T19:58:44.331 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.336+0000 7f9157fff700 1 --2- 172.21.15.53:0/165004094 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91580724d0 0x7f9158070a70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f914800b5f0 tx=0x7f9148006ee0).ready entity=mon.0 client_cookie=98e3be12bc1b12dd server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:44.331 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.336+0000 7f9156ffd700 1 -- 172.21.15.53:0/165004094 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f914800d600 con 0x7f91580724d0 2024-01-21T19:58:44.331 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.336+0000 7f9156ffd700 1 -- 172.21.15.53:0/165004094 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f914800dbd0 con 0x7f91580724d0 2024-01-21T19:58:44.331 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.336+0000 7f9156ffd700 1 -- 172.21.15.53:0/165004094 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9148011c00 con 0x7f91580724d0 2024-01-21T19:58:44.331 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.336+0000 7f915ea44700 1 -- 172.21.15.53:0/165004094 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91580724d0 msgr2=0x7f9158070a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:44.331 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.336+0000 7f915ea44700 1 --2- 172.21.15.53:0/165004094 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91580724d0 0x7f9158070a70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f914800b5f0 tx=0x7f9148006ee0).stop 2024-01-21T19:58:44.331 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.336+0000 7f915ea44700 1 -- 172.21.15.53:0/165004094 shutdown_connections 2024-01-21T19:58:44.332 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.336+0000 7f915ea44700 1 --2- 172.21.15.53:0/165004094 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91580724d0 0x7f9158070a70 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:44.332 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.336+0000 7f915ea44700 1 -- 172.21.15.53:0/165004094 >> 172.21.15.53:0/165004094 conn(0x7f91580f74c0 msgr2=0x7f91580f98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:58:44.332 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.336+0000 7f915ea44700 1 -- 172.21.15.53:0/165004094 shutdown_connections 2024-01-21T19:58:44.332 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.336+0000 7f915ea44700 1 -- 172.21.15.53:0/165004094 wait complete. 2024-01-21T19:58:44.332 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.337+0000 7f915ea44700 1 Processor -- start 2024-01-21T19:58:44.332 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.337+0000 7f915ea44700 1 -- start start 2024-01-21T19:58:44.332 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.337+0000 7f915ea44700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91580724d0 0x7f915810cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:44.332 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.337+0000 7f915ea44700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f915810d1d0 con 0x7f91580724d0 2024-01-21T19:58:44.332 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.338+0000 7f9157fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91580724d0 0x7f915810cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:44.332 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.338+0000 7f9157fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91580724d0 0x7f915810cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49352/0 (socket says 172.21.15.53:49352) 2024-01-21T19:58:44.332 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.338+0000 7f9157fff700 1 -- 172.21.15.53:0/2740811305 learned_addr learned my addr 172.21.15.53:0/2740811305 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:58:44.332 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.338+0000 7f9157fff700 1 -- 172.21.15.53:0/2740811305 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f914800b200 con 0x7f91580724d0 2024-01-21T19:58:44.332 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.338+0000 7f9157fff700 1 --2- 172.21.15.53:0/2740811305 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91580724d0 0x7f915810cce0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f9148011820 tx=0x7f9148006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:44.333 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.338+0000 7f9154ff9700 1 -- 172.21.15.53:0/2740811305 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f914801b410 con 0x7f91580724d0 2024-01-21T19:58:44.333 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.338+0000 7f9154ff9700 1 -- 172.21.15.53:0/2740811305 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f914801b9e0 con 0x7f91580724d0 2024-01-21T19:58:44.333 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.338+0000 7f915ea44700 1 -- 172.21.15.53:0/2740811305 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f915810d3d0 con 0x7f91580724d0 2024-01-21T19:58:44.333 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.338+0000 7f9154ff9700 1 -- 172.21.15.53:0/2740811305 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f914801a8b0 con 0x7f91580724d0 2024-01-21T19:58:44.333 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.338+0000 7f915ea44700 1 -- 172.21.15.53:0/2740811305 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f915810d7e0 con 0x7f91580724d0 2024-01-21T19:58:44.334 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.339+0000 7f9154ff9700 1 -- 172.21.15.53:0/2740811305 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f914800f070 con 0x7f91580724d0 2024-01-21T19:58:44.334 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.339+0000 7f9154ff9700 1 --2- 172.21.15.53:0/2740811305 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7f914402fd50 0x7f9144032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:44.334 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.339+0000 7f9154ff9700 1 -- 172.21.15.53:0/2740811305 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f91480413f0 con 0x7f91580724d0 2024-01-21T19:58:44.334 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.339+0000 7f915ea44700 1 -- 172.21.15.53:0/2740811305 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9158045ad0 con 0x7f91580724d0 2024-01-21T19:58:44.334 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.339+0000 7f91577fe700 1 --2- 172.21.15.53:0/2740811305 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7f914402fd50 0x7f9144032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:44.334 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.340+0000 7f91577fe700 1 --2- 172.21.15.53:0/2740811305 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7f914402fd50 0x7f9144032210 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f914c0096a0 tx=0x7f914c006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:44.335 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.344+0000 7f9154ff9700 1 -- 172.21.15.53:0/2740811305 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9148016070 con 0x7f91580724d0 2024-01-21T19:58:44.494 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.502+0000 7f915ea44700 1 -- 172.21.15.53:0/2740811305 --> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f915810df10 con 0x7f914402fd50 2024-01-21T19:58:44.815 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.824+0000 7f9154ff9700 1 -- 172.21.15.53:0/2740811305 <== mgr.14116 v2:172.21.15.53:6800/55197364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f915810df10 con 0x7f914402fd50 2024-01-21T19:58:44.819 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.825+0000 7f915ea44700 1 -- 172.21.15.53:0/2740811305 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7f914402fd50 msgr2=0x7f9144032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:44.819 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.825+0000 7f915ea44700 1 --2- 172.21.15.53:0/2740811305 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7f914402fd50 0x7f9144032210 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f914c0096a0 tx=0x7f914c006b40).stop 2024-01-21T19:58:44.819 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.825+0000 7f915ea44700 1 -- 172.21.15.53:0/2740811305 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91580724d0 msgr2=0x7f915810cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:44.819 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.825+0000 7f915ea44700 1 --2- 172.21.15.53:0/2740811305 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91580724d0 0x7f915810cce0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f9148011820 tx=0x7f9148006ee0).stop 2024-01-21T19:58:44.820 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.825+0000 7f915ea44700 1 -- 172.21.15.53:0/2740811305 shutdown_connections 2024-01-21T19:58:44.820 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.825+0000 7f915ea44700 1 --2- 172.21.15.53:0/2740811305 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7f914402fd50 0x7f9144032210 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:44.820 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.825+0000 7f915ea44700 1 --2- 172.21.15.53:0/2740811305 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91580724d0 0x7f915810cce0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:44.820 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.825+0000 7f915ea44700 1 -- 172.21.15.53:0/2740811305 >> 172.21.15.53:0/2740811305 conn(0x7f91580f74c0 msgr2=0x7f91580f8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:58:44.820 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.825+0000 7f915ea44700 1 -- 172.21.15.53:0/2740811305 shutdown_connections 2024-01-21T19:58:44.820 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:44.825+0000 7f915ea44700 1 -- 172.21.15.53:0/2740811305 wait complete. 2024-01-21T19:58:45.183 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/57f6722c-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpw7bt2for:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp987f45g0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2024-01-21T19:58:45.552 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.556+0000 7fd52aefa700 1 Processor -- start 2024-01-21T19:58:45.553 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.556+0000 7fd52aefa700 1 -- start start 2024-01-21T19:58:45.553 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.556+0000 7fd52aefa700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5240703b0 0x7fd524100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:45.553 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.556+0000 7fd52aefa700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5240fc5a0 con 0x7fd5240703b0 2024-01-21T19:58:45.553 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.556+0000 7fd529ef8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5240703b0 0x7fd524100bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:45.553 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.556+0000 7fd529ef8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5240703b0 0x7fd524100bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49362/0 (socket says 172.21.15.53:49362) 2024-01-21T19:58:45.553 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.556+0000 7fd529ef8700 1 -- 172.21.15.53:0/1365568122 learned_addr learned my addr 172.21.15.53:0/1365568122 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:58:45.553 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.557+0000 7fd529ef8700 1 -- 172.21.15.53:0/1365568122 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5241010a0 con 0x7fd5240703b0 2024-01-21T19:58:45.553 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.557+0000 7fd529ef8700 1 --2- 172.21.15.53:0/1365568122 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5240703b0 0x7fd524100bb0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fd52000b5f0 tx=0x7fd520006ee0).ready entity=mon.0 client_cookie=f7882ec53d5065a2 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:45.553 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.557+0000 7fd528ef6700 1 -- 172.21.15.53:0/1365568122 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd52000d600 con 0x7fd5240703b0 2024-01-21T19:58:45.553 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.557+0000 7fd528ef6700 1 -- 172.21.15.53:0/1365568122 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd52000dbd0 con 0x7fd5240703b0 2024-01-21T19:58:45.553 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.557+0000 7fd528ef6700 1 -- 172.21.15.53:0/1365568122 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd520011c00 con 0x7fd5240703b0 2024-01-21T19:58:45.553 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.557+0000 7fd52aefa700 1 -- 172.21.15.53:0/1365568122 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5240703b0 msgr2=0x7fd524100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:45.554 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.557+0000 7fd52aefa700 1 --2- 172.21.15.53:0/1365568122 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5240703b0 0x7fd524100bb0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fd52000b5f0 tx=0x7fd520006ee0).stop 2024-01-21T19:58:45.554 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.557+0000 7fd52aefa700 1 -- 172.21.15.53:0/1365568122 shutdown_connections 2024-01-21T19:58:45.554 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.558+0000 7fd52aefa700 1 --2- 172.21.15.53:0/1365568122 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5240703b0 0x7fd524100bb0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:45.554 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.558+0000 7fd52aefa700 1 -- 172.21.15.53:0/1365568122 >> 172.21.15.53:0/1365568122 conn(0x7fd5240f6210 msgr2=0x7fd5240f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:58:45.554 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.558+0000 7fd52aefa700 1 -- 172.21.15.53:0/1365568122 shutdown_connections 2024-01-21T19:58:45.554 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.558+0000 7fd52aefa700 1 -- 172.21.15.53:0/1365568122 wait complete. 2024-01-21T19:58:45.554 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.559+0000 7fd52aefa700 1 Processor -- start 2024-01-21T19:58:45.554 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.559+0000 7fd52aefa700 1 -- start start 2024-01-21T19:58:45.554 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.559+0000 7fd52aefa700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5240703b0 0x7fd52410ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:45.554 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.559+0000 7fd52aefa700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd52410cf60 con 0x7fd5240703b0 2024-01-21T19:58:45.554 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.559+0000 7fd529ef8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5240703b0 0x7fd52410ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:45.554 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.559+0000 7fd529ef8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5240703b0 0x7fd52410ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49372/0 (socket says 172.21.15.53:49372) 2024-01-21T19:58:45.555 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.559+0000 7fd529ef8700 1 -- 172.21.15.53:0/482328989 learned_addr learned my addr 172.21.15.53:0/482328989 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:58:45.555 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.560+0000 7fd529ef8700 1 -- 172.21.15.53:0/482328989 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd52000b200 con 0x7fd5240703b0 2024-01-21T19:58:45.555 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.560+0000 7fd529ef8700 1 --2- 172.21.15.53:0/482328989 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5240703b0 0x7fd52410ca70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fd52000e040 tx=0x7fd520006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:45.555 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.560+0000 7fd5127fc700 1 -- 172.21.15.53:0/482328989 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd52000d9f0 con 0x7fd5240703b0 2024-01-21T19:58:45.555 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.560+0000 7fd52aefa700 1 -- 172.21.15.53:0/482328989 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd52410d160 con 0x7fd5240703b0 2024-01-21T19:58:45.555 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.560+0000 7fd5127fc700 1 -- 172.21.15.53:0/482328989 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd52001a9f0 con 0x7fd5240703b0 2024-01-21T19:58:45.555 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.560+0000 7fd5127fc700 1 -- 172.21.15.53:0/482328989 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd52001b8c0 con 0x7fd5240703b0 2024-01-21T19:58:45.555 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.560+0000 7fd52aefa700 1 -- 172.21.15.53:0/482328989 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd52410d570 con 0x7fd5240703b0 2024-01-21T19:58:45.555 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.561+0000 7fd5127fc700 1 -- 172.21.15.53:0/482328989 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fd52000f070 con 0x7fd5240703b0 2024-01-21T19:58:45.556 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.561+0000 7fd5127fc700 1 --2- 172.21.15.53:0/482328989 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7fd50c02fd50 0x7fd50c032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:45.556 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.561+0000 7fd52aefa700 1 -- 172.21.15.53:0/482328989 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd524045ad0 con 0x7fd5240703b0 2024-01-21T19:58:45.556 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.561+0000 7fd5127fc700 1 -- 172.21.15.53:0/482328989 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fd520041310 con 0x7fd5240703b0 2024-01-21T19:58:45.556 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.561+0000 7fd5296f7700 1 --2- 172.21.15.53:0/482328989 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7fd50c02fd50 0x7fd50c032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:45.556 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.561+0000 7fd5296f7700 1 --2- 172.21.15.53:0/482328989 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7fd50c02fd50 0x7fd50c032210 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fd5140096a0 tx=0x7fd514006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:45.557 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.565+0000 7fd5127fc700 1 -- 172.21.15.53:0/482328989 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd520016070 con 0x7fd5240703b0 2024-01-21T19:58:45.720 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.728+0000 7fd52aefa700 1 -- 172.21.15.53:0/482328989 --> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fd52410de50 con 0x7fd50c02fd50 2024-01-21T19:58:45.720 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.728+0000 7fd5127fc700 1 -- 172.21.15.53:0/482328989 <== mgr.14116 v2:172.21.15.53:6800/55197364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fd52410de50 con 0x7fd50c02fd50 2024-01-21T19:58:45.720 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDIeCV1knv5cd28RNucCPaXPz2KDU4SO6DXjBNlZ1zrk6ikhUc9O/ds7lVcsVom+6z9+NKYnI1K6NrcFC/3mG/F0GEef3/iWlz9NtdLBSA/LODEH+GPnx67D7c6E0e0T+gXEqR/AdUrgjHpqvBlU6yfna+LnUPOaFtPsuUiWn57N/WmPCUJiPkysngopHHsYYiEYVwSTVg1pGbURUGnTJSi1LOBBpgQH05IJR5swYg2QuHuQS5U1X0GmnepSNatC3VozgaxpwfrU0O3h8copBEmOvQS/aIyVilW4EYTJNBQLc+ZD8Va44wbgpstFCOuLKrPHrOBIPUFcrcel960HYECFjjjVqtWEl7saQQI459YvvK0dqBMVXvpI1ZCvfj7rAQXt1qunz6UY8tT+9lrCqL5UE+49zzv9/CpI6W6Bh2eMMkc1ThJxOP62MldS3272FlaO5e1x1BA8cZy3LypxppHv25jiQvRyYkfLfq77a9VE45gvcO8rzTc/he3EXYq7wU= ceph-57f6722c-b897-11ee-95b0-87774f69a715 2024-01-21T19:58:45.723 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.730+0000 7fd52aefa700 1 -- 172.21.15.53:0/482328989 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7fd50c02fd50 msgr2=0x7fd50c032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:45.723 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.730+0000 7fd52aefa700 1 --2- 172.21.15.53:0/482328989 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7fd50c02fd50 0x7fd50c032210 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fd5140096a0 tx=0x7fd514006b40).stop 2024-01-21T19:58:45.723 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.730+0000 7fd52aefa700 1 -- 172.21.15.53:0/482328989 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5240703b0 msgr2=0x7fd52410ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:45.723 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.730+0000 7fd52aefa700 1 --2- 172.21.15.53:0/482328989 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5240703b0 0x7fd52410ca70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fd52000e040 tx=0x7fd520006ee0).stop 2024-01-21T19:58:45.723 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.730+0000 7fd52aefa700 1 -- 172.21.15.53:0/482328989 shutdown_connections 2024-01-21T19:58:45.723 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.730+0000 7fd52aefa700 1 --2- 172.21.15.53:0/482328989 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7fd50c02fd50 0x7fd50c032210 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:45.723 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.730+0000 7fd52aefa700 1 --2- 172.21.15.53:0/482328989 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5240703b0 0x7fd52410ca70 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:45.723 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.730+0000 7fd52aefa700 1 -- 172.21.15.53:0/482328989 >> 172.21.15.53:0/482328989 conn(0x7fd5240f6210 msgr2=0x7fd5240f7d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:58:45.723 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.730+0000 7fd52aefa700 1 -- 172.21.15.53:0/482328989 shutdown_connections 2024-01-21T19:58:45.724 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:45.730+0000 7fd52aefa700 1 -- 172.21.15.53:0/482328989 wait complete. 2024-01-21T19:58:46.091 INFO:teuthology.orchestra.run.smithi053.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-01-21T19:58:46.091 INFO:teuthology.orchestra.run.smithi053.stderr:Adding key to root@localhost authorized_keys... 2024-01-21T19:58:46.092 INFO:teuthology.orchestra.run.smithi053.stderr:Adding host smithi053... 2024-01-21T19:58:46.092 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/57f6722c-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpw7bt2for:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp987f45g0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi053 172.21.15.53 2024-01-21T19:58:46.184 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:45 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cephadm 2024-01-21T19:58:44.504402+0000 mgr.smithi053.fgsmol (mgr.14116) 6 : cephadm [INF] Generating ssh key... 2024-01-21T19:58:46.184 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:45 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:44.822740+0000 mon.smithi053 (mon.0) 53 : audit [INF] from='mgr.14116 172.21.15.53:0/1503870519' entity='mgr.smithi053.fgsmol' 2024-01-21T19:58:46.184 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:45 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:44.824438+0000 mon.smithi053 (mon.0) 54 : audit [INF] from='mgr.14116 172.21.15.53:0/1503870519' entity='mgr.smithi053.fgsmol' 2024-01-21T19:58:46.414 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:46.422+0000 7f63983bf700 1 Processor -- start 2024-01-21T19:58:46.420 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:46.422+0000 7f63983bf700 1 -- start start 2024-01-21T19:58:46.420 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:46.422+0000 7f63983bf700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6390070390 0x7f6390100d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:46.420 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:46.422+0000 7f63983bf700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63900fc770 con 0x7f6390070390 2024-01-21T19:58:46.421 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:46.423+0000 7f639615b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6390070390 0x7f6390100d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:46.421 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:46.423+0000 7f639615b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6390070390 0x7f6390100d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49386/0 (socket says 172.21.15.53:49386) 2024-01-21T19:58:46.421 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:46.423+0000 7f639615b700 1 -- 172.21.15.53:0/929313098 learned_addr learned my addr 172.21.15.53:0/929313098 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:58:46.421 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:46.423+0000 7f639615b700 1 -- 172.21.15.53:0/929313098 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6390101270 con 0x7f6390070390 2024-01-21T19:58:46.421 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:46.423+0000 7f639615b700 1 --2- 172.21.15.53:0/929313098 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6390070390 0x7f6390100d80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f638c00b5f0 tx=0x7f638c006ee0).ready entity=mon.0 client_cookie=8d2d62a1089941e5 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:46.421 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:46.423+0000 7f6395159700 1 -- 172.21.15.53:0/929313098 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f638c00d600 con 0x7f6390070390 2024-01-21T19:58:46.421 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:46.423+0000 7f6395159700 1 -- 172.21.15.53:0/929313098 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f638c00dbd0 con 0x7f6390070390 2024-01-21T19:58:46.421 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:46.423+0000 7f6395159700 1 -- 172.21.15.53:0/929313098 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f638c011c00 con 0x7f6390070390 2024-01-21T19:58:46.421 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:46.424+0000 7f63983bf700 1 -- 172.21.15.53:0/929313098 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6390070390 msgr2=0x7f6390100d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:46.421 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:46.424+0000 7f63983bf700 1 --2- 172.21.15.53:0/929313098 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6390070390 0x7f6390100d80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f638c00b5f0 tx=0x7f638c006ee0).stop 2024-01-21T19:58:46.422 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:46.424+0000 7f63983bf700 1 -- 172.21.15.53:0/929313098 shutdown_connections 2024-01-21T19:58:46.422 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:46.424+0000 7f63983bf700 1 --2- 172.21.15.53:0/929313098 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6390070390 0x7f6390100d80 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:46.422 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:46.424+0000 7f63983bf700 1 -- 172.21.15.53:0/929313098 >> 172.21.15.53:0/929313098 conn(0x7f63900f63c0 msgr2=0x7f63900f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:58:46.422 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:46.424+0000 7f63983bf700 1 -- 172.21.15.53:0/929313098 shutdown_connections 2024-01-21T19:58:46.422 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:46.424+0000 7f63983bf700 1 -- 172.21.15.53:0/929313098 wait complete. 2024-01-21T19:58:46.422 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:46.425+0000 7f63983bf700 1 Processor -- start 2024-01-21T19:58:46.422 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:46.425+0000 7f63983bf700 1 -- start start 2024-01-21T19:58:46.422 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:46.425+0000 7f63983bf700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6390070390 0x7f639010cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:46.422 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:46.425+0000 7f63983bf700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f639010d0f0 con 0x7f6390070390 2024-01-21T19:58:46.423 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:46.425+0000 7f639615b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6390070390 0x7f639010cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:46.423 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:46.425+0000 7f639615b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6390070390 0x7f639010cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49398/0 (socket says 172.21.15.53:49398) 2024-01-21T19:58:46.423 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:46.425+0000 7f639615b700 1 -- 172.21.15.53:0/169112549 learned_addr learned my addr 172.21.15.53:0/169112549 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:58:46.423 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:46.425+0000 7f639615b700 1 -- 172.21.15.53:0/169112549 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f638c00b200 con 0x7f6390070390 2024-01-21T19:58:46.423 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:46.425+0000 7f639615b700 1 --2- 172.21.15.53:0/169112549 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6390070390 0x7f639010cc00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f638c00b190 tx=0x7f638c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:46.423 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:46.426+0000 7f637effd700 1 -- 172.21.15.53:0/169112549 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f638c01b410 con 0x7f6390070390 2024-01-21T19:58:46.423 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:46.426+0000 7f63983bf700 1 -- 172.21.15.53:0/169112549 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f639010d2f0 con 0x7f6390070390 2024-01-21T19:58:46.423 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:46.426+0000 7f637effd700 1 -- 172.21.15.53:0/169112549 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f638c01b9e0 con 0x7f6390070390 2024-01-21T19:58:46.423 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:46.426+0000 7f637effd700 1 -- 172.21.15.53:0/169112549 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f638c01a880 con 0x7f6390070390 2024-01-21T19:58:46.424 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:46.426+0000 7f63983bf700 1 -- 172.21.15.53:0/169112549 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f639010d700 con 0x7f6390070390 2024-01-21T19:58:46.424 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:46.426+0000 7f637effd700 1 -- 172.21.15.53:0/169112549 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f638c00f070 con 0x7f6390070390 2024-01-21T19:58:46.424 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:46.426+0000 7f637effd700 1 --2- 172.21.15.53:0/169112549 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7f637802fd50 0x7f6378032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:46.424 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:46.427+0000 7f637effd700 1 -- 172.21.15.53:0/169112549 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f638c042140 con 0x7f6390070390 2024-01-21T19:58:46.424 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:46.427+0000 7f639595a700 1 --2- 172.21.15.53:0/169112549 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7f637802fd50 0x7f6378032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:46.424 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:46.427+0000 7f639595a700 1 --2- 172.21.15.53:0/169112549 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7f637802fd50 0x7f6378032210 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f63800096a0 tx=0x7f6380006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:46.424 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:46.428+0000 7f63983bf700 1 -- 172.21.15.53:0/169112549 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6370004fa0 con 0x7f6390070390 2024-01-21T19:58:46.425 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:46.432+0000 7f637effd700 1 -- 172.21.15.53:0/169112549 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f638c010360 con 0x7f6390070390 2024-01-21T19:58:46.604 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:46.612+0000 7f63983bf700 1 -- 172.21.15.53:0/169112549 --> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi053", "addr": "172.21.15.53", "target": ["mon-mgr", ""]}) v1 -- 0x7f63700029f0 con 0x7f637802fd50 2024-01-21T19:58:47.200 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: Added host 'smithi053' 2024-01-21T19:58:47.200 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:47.207+0000 7f637effd700 1 -- 172.21.15.53:0/169112549 <== mgr.14116 v2:172.21.15.53:6800/55197364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f63700029f0 con 0x7f637802fd50 2024-01-21T19:58:47.202 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:47.209+0000 7f63983bf700 1 -- 172.21.15.53:0/169112549 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7f637802fd50 msgr2=0x7f6378032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:47.202 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:47.209+0000 7f63983bf700 1 --2- 172.21.15.53:0/169112549 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7f637802fd50 0x7f6378032210 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f63800096a0 tx=0x7f6380006b40).stop 2024-01-21T19:58:47.202 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:47.209+0000 7f63983bf700 1 -- 172.21.15.53:0/169112549 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6390070390 msgr2=0x7f639010cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:47.202 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:47.209+0000 7f63983bf700 1 --2- 172.21.15.53:0/169112549 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6390070390 0x7f639010cc00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f638c00b190 tx=0x7f638c006ee0).stop 2024-01-21T19:58:47.202 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:47.209+0000 7f63983bf700 1 -- 172.21.15.53:0/169112549 shutdown_connections 2024-01-21T19:58:47.202 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:47.209+0000 7f63983bf700 1 --2- 172.21.15.53:0/169112549 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7f637802fd50 0x7f6378032210 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:47.202 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:47.209+0000 7f63983bf700 1 --2- 172.21.15.53:0/169112549 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6390070390 0x7f639010cc00 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:47.202 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:47.209+0000 7f63983bf700 1 -- 172.21.15.53:0/169112549 >> 172.21.15.53:0/169112549 conn(0x7f63900f63c0 msgr2=0x7f63900f7ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:58:47.202 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:47.209+0000 7f63983bf700 1 -- 172.21.15.53:0/169112549 shutdown_connections 2024-01-21T19:58:47.202 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:47.209+0000 7f63983bf700 1 -- 172.21.15.53:0/169112549 wait complete. 2024-01-21T19:58:47.560 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:47 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:45.729938+0000 mgr.smithi053.fgsmol (mgr.14116) 7 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T19:58:47.560 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:47 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:47.207449+0000 mon.smithi053 (mon.0) 55 : audit [INF] from='mgr.14116 172.21.15.53:0/1503870519' entity='mgr.smithi053.fgsmol' 2024-01-21T19:58:47.560 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:47 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:47.207995+0000 mon.smithi053 (mon.0) 56 : audit [DBG] from='mgr.14116 172.21.15.53:0/1503870519' entity='mgr.smithi053.fgsmol' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T19:58:47.577 INFO:teuthology.orchestra.run.smithi053.stderr:Deploying mon service with default placement... 2024-01-21T19:58:47.578 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/57f6722c-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpw7bt2for:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp987f45g0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2024-01-21T19:58:48.077 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.081+0000 7f5716686700 1 Processor -- start 2024-01-21T19:58:48.077 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.081+0000 7f5716686700 1 -- start start 2024-01-21T19:58:48.077 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.081+0000 7f5716686700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5710070370 0x7f57100fe9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:48.077 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.081+0000 7f5716686700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5710070790 con 0x7f5710070370 2024-01-21T19:58:48.077 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.082+0000 7f570ffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5710070370 0x7f57100fe9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:48.077 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.082+0000 7f570ffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5710070370 0x7f57100fe9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47866/0 (socket says 172.21.15.53:47866) 2024-01-21T19:58:48.077 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.082+0000 7f570ffff700 1 -- 172.21.15.53:0/3992901976 learned_addr learned my addr 172.21.15.53:0/3992901976 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:58:48.077 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.082+0000 7f570ffff700 1 -- 172.21.15.53:0/3992901976 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57100fee90 con 0x7f5710070370 2024-01-21T19:58:48.077 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.082+0000 7f570ffff700 1 --2- 172.21.15.53:0/3992901976 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5710070370 0x7f57100fe9a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f56f800b5f0 tx=0x7f56f8006ee0).ready entity=mon.0 client_cookie=ab46c301b840dfcf server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:48.077 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.082+0000 7f570effd700 1 -- 172.21.15.53:0/3992901976 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56f800d600 con 0x7f5710070370 2024-01-21T19:58:48.078 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.082+0000 7f570effd700 1 -- 172.21.15.53:0/3992901976 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f56f800dbd0 con 0x7f5710070370 2024-01-21T19:58:48.078 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.082+0000 7f570effd700 1 -- 172.21.15.53:0/3992901976 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56f8011c00 con 0x7f5710070370 2024-01-21T19:58:48.078 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.083+0000 7f5716686700 1 -- 172.21.15.53:0/3992901976 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5710070370 msgr2=0x7f57100fe9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:48.078 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.083+0000 7f5716686700 1 --2- 172.21.15.53:0/3992901976 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5710070370 0x7f57100fe9a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f56f800b5f0 tx=0x7f56f8006ee0).stop 2024-01-21T19:58:48.078 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.083+0000 7f5716686700 1 -- 172.21.15.53:0/3992901976 shutdown_connections 2024-01-21T19:58:48.078 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.083+0000 7f5716686700 1 --2- 172.21.15.53:0/3992901976 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5710070370 0x7f57100fe9a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:48.078 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.083+0000 7f5716686700 1 -- 172.21.15.53:0/3992901976 >> 172.21.15.53:0/3992901976 conn(0x7f57100f63c0 msgr2=0x7f57100f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:58:48.078 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.083+0000 7f5716686700 1 -- 172.21.15.53:0/3992901976 shutdown_connections 2024-01-21T19:58:48.078 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.083+0000 7f5716686700 1 -- 172.21.15.53:0/3992901976 wait complete. 2024-01-21T19:58:48.078 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.083+0000 7f5716686700 1 Processor -- start 2024-01-21T19:58:48.078 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.084+0000 7f5716686700 1 -- start start 2024-01-21T19:58:48.078 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.084+0000 7f5716686700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5710070370 0x7f571010a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:48.078 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.084+0000 7f5716686700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f571010aea0 con 0x7f5710070370 2024-01-21T19:58:48.078 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.084+0000 7f570ffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5710070370 0x7f571010a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:48.078 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.084+0000 7f570ffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5710070370 0x7f571010a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47878/0 (socket says 172.21.15.53:47878) 2024-01-21T19:58:48.079 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.084+0000 7f570ffff700 1 -- 172.21.15.53:0/3821291622 learned_addr learned my addr 172.21.15.53:0/3821291622 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:58:48.079 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.084+0000 7f570ffff700 1 -- 172.21.15.53:0/3821291622 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56f800b200 con 0x7f5710070370 2024-01-21T19:58:48.079 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.084+0000 7f570ffff700 1 --2- 172.21.15.53:0/3821291622 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5710070370 0x7f571010a9b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f56f800b190 tx=0x7f56f8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:48.079 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.085+0000 7f570cff9700 1 -- 172.21.15.53:0/3821291622 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56f801b410 con 0x7f5710070370 2024-01-21T19:58:48.079 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.085+0000 7f570cff9700 1 -- 172.21.15.53:0/3821291622 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f56f801b9e0 con 0x7f5710070370 2024-01-21T19:58:48.079 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.085+0000 7f5716686700 1 -- 172.21.15.53:0/3821291622 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f571010b0a0 con 0x7f5710070370 2024-01-21T19:58:48.079 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.085+0000 7f570cff9700 1 -- 172.21.15.53:0/3821291622 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56f801a8f0 con 0x7f5710070370 2024-01-21T19:58:48.079 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.085+0000 7f5716686700 1 -- 172.21.15.53:0/3821291622 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f571010b4b0 con 0x7f5710070370 2024-01-21T19:58:48.080 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.086+0000 7f570cff9700 1 -- 172.21.15.53:0/3821291622 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f56f800f070 con 0x7f5710070370 2024-01-21T19:58:48.080 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.086+0000 7f5716686700 1 -- 172.21.15.53:0/3821291622 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5710045ad0 con 0x7f5710070370 2024-01-21T19:58:48.080 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.086+0000 7f570cff9700 1 --2- 172.21.15.53:0/3821291622 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7f56f002fd50 0x7f56f0032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:48.080 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.086+0000 7f570cff9700 1 -- 172.21.15.53:0/3821291622 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f56f8041380 con 0x7f5710070370 2024-01-21T19:58:48.080 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.086+0000 7f570f7fe700 1 --2- 172.21.15.53:0/3821291622 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7f56f002fd50 0x7f56f0032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:48.080 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.086+0000 7f570f7fe700 1 --2- 172.21.15.53:0/3821291622 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7f56f002fd50 0x7f56f0032210 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f57000096a0 tx=0x7f5700006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:48.082 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.090+0000 7f570cff9700 1 -- 172.21.15.53:0/3821291622 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f56f8019af0 con 0x7f5710070370 2024-01-21T19:58:48.267 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.275+0000 7f5716686700 1 -- 172.21.15.53:0/3821291622 --> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f571010bd90 con 0x7f56f002fd50 2024-01-21T19:58:48.271 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.279+0000 7f570cff9700 1 -- 172.21.15.53:0/3821291622 <== mgr.14116 v2:172.21.15.53:6800/55197364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f571010bd90 con 0x7f56f002fd50 2024-01-21T19:58:48.271 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: Scheduled mon update... 2024-01-21T19:58:48.274 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.281+0000 7f5716686700 1 -- 172.21.15.53:0/3821291622 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7f56f002fd50 msgr2=0x7f56f0032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:48.274 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.281+0000 7f5716686700 1 --2- 172.21.15.53:0/3821291622 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7f56f002fd50 0x7f56f0032210 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f57000096a0 tx=0x7f5700006b40).stop 2024-01-21T19:58:48.274 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.281+0000 7f5716686700 1 -- 172.21.15.53:0/3821291622 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5710070370 msgr2=0x7f571010a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:48.274 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.281+0000 7f5716686700 1 --2- 172.21.15.53:0/3821291622 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5710070370 0x7f571010a9b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f56f800b190 tx=0x7f56f8006ee0).stop 2024-01-21T19:58:48.274 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.281+0000 7f5716686700 1 -- 172.21.15.53:0/3821291622 shutdown_connections 2024-01-21T19:58:48.274 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.281+0000 7f5716686700 1 --2- 172.21.15.53:0/3821291622 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7f56f002fd50 0x7f56f0032210 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:48.274 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.281+0000 7f5716686700 1 --2- 172.21.15.53:0/3821291622 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5710070370 0x7f571010a9b0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:48.274 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.281+0000 7f5716686700 1 -- 172.21.15.53:0/3821291622 >> 172.21.15.53:0/3821291622 conn(0x7f57100f63c0 msgr2=0x7f57100f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:58:48.274 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.281+0000 7f5716686700 1 -- 172.21.15.53:0/3821291622 shutdown_connections 2024-01-21T19:58:48.274 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:48.281+0000 7f5716686700 1 -- 172.21.15.53:0/3821291622 wait complete. 2024-01-21T19:58:48.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:48 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:46.614746+0000 mgr.smithi053.fgsmol (mgr.14116) 8 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi053", "addr": "172.21.15.53", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T19:58:48.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:48 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cephadm 2024-01-21T19:58:47.207759+0000 mgr.smithi053.fgsmol (mgr.14116) 9 : cephadm [INF] Added host smithi053 2024-01-21T19:58:48.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:48 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:48.280632+0000 mon.smithi053 (mon.0) 57 : audit [INF] from='mgr.14116 172.21.15.53:0/1503870519' entity='mgr.smithi053.fgsmol' 2024-01-21T19:58:48.710 INFO:teuthology.orchestra.run.smithi053.stderr:Deploying mgr service with default placement... 2024-01-21T19:58:48.710 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/57f6722c-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpw7bt2for:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp987f45g0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2024-01-21T19:58:49.047 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.055+0000 7fa8192e3700 1 Processor -- start 2024-01-21T19:58:49.047 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.055+0000 7fa8192e3700 1 -- start start 2024-01-21T19:58:49.048 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.055+0000 7fa8192e3700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa814070350 0x7fa8140fe840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:49.048 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.055+0000 7fa8192e3700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa814070770 con 0x7fa814070350 2024-01-21T19:58:49.048 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.055+0000 7fa812ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa814070350 0x7fa8140fe840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:49.048 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.055+0000 7fa812ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa814070350 0x7fa8140fe840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47882/0 (socket says 172.21.15.53:47882) 2024-01-21T19:58:49.048 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.055+0000 7fa812ffd700 1 -- 172.21.15.53:0/3006055150 learned_addr learned my addr 172.21.15.53:0/3006055150 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:58:49.050 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.055+0000 7fa812ffd700 1 -- 172.21.15.53:0/3006055150 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8140fed30 con 0x7fa814070350 2024-01-21T19:58:49.050 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.056+0000 7fa812ffd700 1 --2- 172.21.15.53:0/3006055150 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa814070350 0x7fa8140fe840 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fa80400b5f0 tx=0x7fa804006ee0).ready entity=mon.0 client_cookie=1dc3b869c1f1ecd3 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:49.050 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.056+0000 7fa811ffb700 1 -- 172.21.15.53:0/3006055150 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa80400d600 con 0x7fa814070350 2024-01-21T19:58:49.050 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.056+0000 7fa811ffb700 1 -- 172.21.15.53:0/3006055150 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa80400dbd0 con 0x7fa814070350 2024-01-21T19:58:49.050 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.056+0000 7fa811ffb700 1 -- 172.21.15.53:0/3006055150 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa804011c00 con 0x7fa814070350 2024-01-21T19:58:49.050 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.057+0000 7fa8192e3700 1 -- 172.21.15.53:0/3006055150 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa814070350 msgr2=0x7fa8140fe840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:49.050 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.057+0000 7fa8192e3700 1 --2- 172.21.15.53:0/3006055150 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa814070350 0x7fa8140fe840 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fa80400b5f0 tx=0x7fa804006ee0).stop 2024-01-21T19:58:49.050 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.057+0000 7fa8192e3700 1 -- 172.21.15.53:0/3006055150 shutdown_connections 2024-01-21T19:58:49.050 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.057+0000 7fa8192e3700 1 --2- 172.21.15.53:0/3006055150 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa814070350 0x7fa8140fe840 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:49.051 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.057+0000 7fa8192e3700 1 -- 172.21.15.53:0/3006055150 >> 172.21.15.53:0/3006055150 conn(0x7fa8140f6240 msgr2=0x7fa8140f8660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:58:49.051 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.057+0000 7fa8192e3700 1 -- 172.21.15.53:0/3006055150 shutdown_connections 2024-01-21T19:58:49.051 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.057+0000 7fa8192e3700 1 -- 172.21.15.53:0/3006055150 wait complete. 2024-01-21T19:58:49.051 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.058+0000 7fa8192e3700 1 Processor -- start 2024-01-21T19:58:49.051 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.058+0000 7fa8192e3700 1 -- start start 2024-01-21T19:58:49.051 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.058+0000 7fa8192e3700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa814070350 0x7fa8140fc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:49.051 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.058+0000 7fa8192e3700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8140fa740 con 0x7fa814070350 2024-01-21T19:58:49.053 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.058+0000 7fa812ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa814070350 0x7fa8140fc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:49.053 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.058+0000 7fa812ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa814070350 0x7fa8140fc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47898/0 (socket says 172.21.15.53:47898) 2024-01-21T19:58:49.053 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.058+0000 7fa812ffd700 1 -- 172.21.15.53:0/181516220 learned_addr learned my addr 172.21.15.53:0/181516220 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:58:49.053 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.059+0000 7fa812ffd700 1 -- 172.21.15.53:0/181516220 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa80400b200 con 0x7fa814070350 2024-01-21T19:58:49.053 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.059+0000 7fa812ffd700 1 --2- 172.21.15.53:0/181516220 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa814070350 0x7fa8140fc2c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fa80400b190 tx=0x7fa804006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:49.053 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.059+0000 7fa7fb7fe700 1 -- 172.21.15.53:0/181516220 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa80401b410 con 0x7fa814070350 2024-01-21T19:58:49.053 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.059+0000 7fa8192e3700 1 -- 172.21.15.53:0/181516220 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8140fa8e0 con 0x7fa814070350 2024-01-21T19:58:49.053 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.059+0000 7fa7fb7fe700 1 -- 172.21.15.53:0/181516220 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa80401b9e0 con 0x7fa814070350 2024-01-21T19:58:49.053 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.059+0000 7fa7fb7fe700 1 -- 172.21.15.53:0/181516220 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa80401a8f0 con 0x7fa814070350 2024-01-21T19:58:49.053 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.059+0000 7fa8192e3700 1 -- 172.21.15.53:0/181516220 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8140facf0 con 0x7fa814070350 2024-01-21T19:58:49.053 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.060+0000 7fa7fb7fe700 1 -- 172.21.15.53:0/181516220 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fa80400f070 con 0x7fa814070350 2024-01-21T19:58:49.053 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.060+0000 7fa7fb7fe700 1 --2- 172.21.15.53:0/181516220 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7fa80002fd50 0x7fa800032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:49.053 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.060+0000 7fa7fb7fe700 1 -- 172.21.15.53:0/181516220 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fa804041380 con 0x7fa814070350 2024-01-21T19:58:49.054 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.060+0000 7fa8127fc700 1 --2- 172.21.15.53:0/181516220 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7fa80002fd50 0x7fa800032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:49.054 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.060+0000 7fa8192e3700 1 -- 172.21.15.53:0/181516220 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa814045ad0 con 0x7fa814070350 2024-01-21T19:58:49.054 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.060+0000 7fa8127fc700 1 --2- 172.21.15.53:0/181516220 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7fa80002fd50 0x7fa800032210 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fa8080096a0 tx=0x7fa808006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:49.056 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.065+0000 7fa7fb7fe700 1 -- 172.21.15.53:0/181516220 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa804016070 con 0x7fa814070350 2024-01-21T19:58:49.243 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.251+0000 7fa8192e3700 1 -- 172.21.15.53:0/181516220 --> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7fa8140ffa20 con 0x7fa80002fd50 2024-01-21T19:58:49.246 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.254+0000 7fa7fb7fe700 1 -- 172.21.15.53:0/181516220 <== mgr.14116 v2:172.21.15.53:6800/55197364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fa8140ffa20 con 0x7fa80002fd50 2024-01-21T19:58:49.246 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-01-21T19:58:49.249 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.256+0000 7fa8192e3700 1 -- 172.21.15.53:0/181516220 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7fa80002fd50 msgr2=0x7fa800032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:49.249 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.256+0000 7fa8192e3700 1 --2- 172.21.15.53:0/181516220 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7fa80002fd50 0x7fa800032210 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fa8080096a0 tx=0x7fa808006b40).stop 2024-01-21T19:58:49.249 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.256+0000 7fa8192e3700 1 -- 172.21.15.53:0/181516220 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa814070350 msgr2=0x7fa8140fc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:49.249 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.256+0000 7fa8192e3700 1 --2- 172.21.15.53:0/181516220 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa814070350 0x7fa8140fc2c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fa80400b190 tx=0x7fa804006ee0).stop 2024-01-21T19:58:49.249 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.256+0000 7fa8192e3700 1 -- 172.21.15.53:0/181516220 shutdown_connections 2024-01-21T19:58:49.249 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.256+0000 7fa8192e3700 1 --2- 172.21.15.53:0/181516220 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7fa80002fd50 0x7fa800032210 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:49.249 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.256+0000 7fa8192e3700 1 --2- 172.21.15.53:0/181516220 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa814070350 0x7fa8140fc2c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:49.249 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.256+0000 7fa8192e3700 1 -- 172.21.15.53:0/181516220 >> 172.21.15.53:0/181516220 conn(0x7fa8140f6240 msgr2=0x7fa8140f7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:58:49.249 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.256+0000 7fa8192e3700 1 -- 172.21.15.53:0/181516220 shutdown_connections 2024-01-21T19:58:49.249 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:49.256+0000 7fa8192e3700 1 -- 172.21.15.53:0/181516220 wait complete. 2024-01-21T19:58:49.543 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:49 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:48.277697+0000 mgr.smithi053.fgsmol (mgr.14116) 10 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T19:58:49.543 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:49 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cephadm 2024-01-21T19:58:48.278477+0000 mgr.smithi053.fgsmol (mgr.14116) 11 : cephadm [INF] Saving service mon spec with placement count:5 2024-01-21T19:58:49.543 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:49 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:49.255736+0000 mon.smithi053 (mon.0) 58 : audit [INF] from='mgr.14116 172.21.15.53:0/1503870519' entity='mgr.smithi053.fgsmol' 2024-01-21T19:58:49.624 INFO:teuthology.orchestra.run.smithi053.stderr:Deploying crash service with default placement... 2024-01-21T19:58:49.624 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/57f6722c-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpw7bt2for:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp987f45g0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2024-01-21T19:58:50.034 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.040+0000 7f592e2d8700 1 Processor -- start 2024-01-21T19:58:50.034 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.040+0000 7f592e2d8700 1 -- start start 2024-01-21T19:58:50.034 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.040+0000 7f592e2d8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5928070460 0x7f59280fe930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:50.034 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.040+0000 7f592e2d8700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59280fee20 con 0x7f5928070460 2024-01-21T19:58:50.034 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.040+0000 7f5927fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5928070460 0x7f59280fe930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:50.034 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.040+0000 7f5927fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5928070460 0x7f59280fe930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47910/0 (socket says 172.21.15.53:47910) 2024-01-21T19:58:50.034 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.040+0000 7f5927fff700 1 -- 172.21.15.53:0/977597866 learned_addr learned my addr 172.21.15.53:0/977597866 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:58:50.034 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.040+0000 7f5927fff700 1 -- 172.21.15.53:0/977597866 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59280fef60 con 0x7f5928070460 2024-01-21T19:58:50.035 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.040+0000 7f5927fff700 1 --2- 172.21.15.53:0/977597866 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5928070460 0x7f59280fe930 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f591800b5f0 tx=0x7f5918006ee0).ready entity=mon.0 client_cookie=49788136e6f5d0dc server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:50.035 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.041+0000 7f5926ffd700 1 -- 172.21.15.53:0/977597866 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f591800d600 con 0x7f5928070460 2024-01-21T19:58:50.035 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.041+0000 7f5926ffd700 1 -- 172.21.15.53:0/977597866 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f591800dbd0 con 0x7f5928070460 2024-01-21T19:58:50.035 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.041+0000 7f5926ffd700 1 -- 172.21.15.53:0/977597866 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5918011c00 con 0x7f5928070460 2024-01-21T19:58:50.035 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.041+0000 7f592e2d8700 1 -- 172.21.15.53:0/977597866 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5928070460 msgr2=0x7f59280fe930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:50.035 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.041+0000 7f592e2d8700 1 --2- 172.21.15.53:0/977597866 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5928070460 0x7f59280fe930 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f591800b5f0 tx=0x7f5918006ee0).stop 2024-01-21T19:58:50.035 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.041+0000 7f592e2d8700 1 -- 172.21.15.53:0/977597866 shutdown_connections 2024-01-21T19:58:50.035 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.041+0000 7f592e2d8700 1 --2- 172.21.15.53:0/977597866 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5928070460 0x7f59280fe930 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:50.035 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.041+0000 7f592e2d8700 1 -- 172.21.15.53:0/977597866 >> 172.21.15.53:0/977597866 conn(0x7f59280f63b0 msgr2=0x7f59280f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:58:50.035 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.041+0000 7f592e2d8700 1 -- 172.21.15.53:0/977597866 shutdown_connections 2024-01-21T19:58:50.035 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.041+0000 7f592e2d8700 1 -- 172.21.15.53:0/977597866 wait complete. 2024-01-21T19:58:50.035 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.042+0000 7f592e2d8700 1 Processor -- start 2024-01-21T19:58:50.035 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.042+0000 7f592e2d8700 1 -- start start 2024-01-21T19:58:50.035 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.042+0000 7f592e2d8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5928070460 0x7f59280fc3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:50.035 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.042+0000 7f592e2d8700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59280fa830 con 0x7f5928070460 2024-01-21T19:58:50.037 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.042+0000 7f5927fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5928070460 0x7f59280fc3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:50.037 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.042+0000 7f5927fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5928070460 0x7f59280fc3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47922/0 (socket says 172.21.15.53:47922) 2024-01-21T19:58:50.038 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.042+0000 7f5927fff700 1 -- 172.21.15.53:0/2061699617 learned_addr learned my addr 172.21.15.53:0/2061699617 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:58:50.038 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.043+0000 7f5927fff700 1 -- 172.21.15.53:0/2061699617 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f591800b200 con 0x7f5928070460 2024-01-21T19:58:50.038 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.043+0000 7f5927fff700 1 --2- 172.21.15.53:0/2061699617 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5928070460 0x7f59280fc3b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f591800b190 tx=0x7f5918006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:50.038 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.043+0000 7f5924ff9700 1 -- 172.21.15.53:0/2061699617 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f591801b410 con 0x7f5928070460 2024-01-21T19:58:50.038 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.043+0000 7f5924ff9700 1 -- 172.21.15.53:0/2061699617 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f591801b9e0 con 0x7f5928070460 2024-01-21T19:58:50.038 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.043+0000 7f592e2d8700 1 -- 172.21.15.53:0/2061699617 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59280fa9d0 con 0x7f5928070460 2024-01-21T19:58:50.038 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.043+0000 7f5924ff9700 1 -- 172.21.15.53:0/2061699617 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f591801a880 con 0x7f5928070460 2024-01-21T19:58:50.038 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.043+0000 7f592e2d8700 1 -- 172.21.15.53:0/2061699617 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59280fade0 con 0x7f5928070460 2024-01-21T19:58:50.038 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.044+0000 7f592e2d8700 1 -- 172.21.15.53:0/2061699617 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5928045ad0 con 0x7f5928070460 2024-01-21T19:58:50.038 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.044+0000 7f5924ff9700 1 -- 172.21.15.53:0/2061699617 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f591800f070 con 0x7f5928070460 2024-01-21T19:58:50.038 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.045+0000 7f5924ff9700 1 --2- 172.21.15.53:0/2061699617 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7f591402fdb0 0x7f5914032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:50.038 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.045+0000 7f5924ff9700 1 -- 172.21.15.53:0/2061699617 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f5918041d50 con 0x7f5928070460 2024-01-21T19:58:50.038 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.045+0000 7f59277fe700 1 --2- 172.21.15.53:0/2061699617 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7f591402fdb0 0x7f5914032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:50.039 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.045+0000 7f59277fe700 1 --2- 172.21.15.53:0/2061699617 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7f591402fdb0 0x7f5914032270 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f591c0096a0 tx=0x7f591c006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:50.040 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.048+0000 7f5924ff9700 1 -- 172.21.15.53:0/2061699617 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f591802db70 con 0x7f5928070460 2024-01-21T19:58:50.225 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.233+0000 7f592e2d8700 1 -- 172.21.15.53:0/2061699617 --> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f59280fb020 con 0x7f591402fdb0 2024-01-21T19:58:50.228 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.236+0000 7f5924ff9700 1 -- 172.21.15.53:0/2061699617 <== mgr.14116 v2:172.21.15.53:6800/55197364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f59280fb020 con 0x7f591402fdb0 2024-01-21T19:58:50.228 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: Scheduled crash update... 2024-01-21T19:58:50.231 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.238+0000 7f592e2d8700 1 -- 172.21.15.53:0/2061699617 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7f591402fdb0 msgr2=0x7f5914032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:50.231 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.238+0000 7f592e2d8700 1 --2- 172.21.15.53:0/2061699617 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7f591402fdb0 0x7f5914032270 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f591c0096a0 tx=0x7f591c006b40).stop 2024-01-21T19:58:50.231 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.238+0000 7f592e2d8700 1 -- 172.21.15.53:0/2061699617 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5928070460 msgr2=0x7f59280fc3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:50.231 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.238+0000 7f592e2d8700 1 --2- 172.21.15.53:0/2061699617 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5928070460 0x7f59280fc3b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f591800b190 tx=0x7f5918006ee0).stop 2024-01-21T19:58:50.231 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.238+0000 7f592e2d8700 1 -- 172.21.15.53:0/2061699617 shutdown_connections 2024-01-21T19:58:50.231 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.238+0000 7f592e2d8700 1 --2- 172.21.15.53:0/2061699617 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7f591402fdb0 0x7f5914032270 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:50.231 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.238+0000 7f592e2d8700 1 --2- 172.21.15.53:0/2061699617 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5928070460 0x7f59280fc3b0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:50.231 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.238+0000 7f592e2d8700 1 -- 172.21.15.53:0/2061699617 >> 172.21.15.53:0/2061699617 conn(0x7f59280f63b0 msgr2=0x7f59280f7f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:58:50.231 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.238+0000 7f592e2d8700 1 -- 172.21.15.53:0/2061699617 shutdown_connections 2024-01-21T19:58:50.232 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:50.238+0000 7f592e2d8700 1 -- 172.21.15.53:0/2061699617 wait complete. 2024-01-21T19:58:50.601 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:50 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:49.253147+0000 mgr.smithi053.fgsmol (mgr.14116) 12 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T19:58:50.601 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:50 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cephadm 2024-01-21T19:58:49.253674+0000 mgr.smithi053.fgsmol (mgr.14116) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2024-01-21T19:58:50.601 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:50 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:50.237743+0000 mon.smithi053 (mon.0) 59 : audit [INF] from='mgr.14116 172.21.15.53:0/1503870519' entity='mgr.smithi053.fgsmol' 2024-01-21T19:58:50.707 INFO:teuthology.orchestra.run.smithi053.stderr:Enabling mgr prometheus module... 2024-01-21T19:58:50.707 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/57f6722c-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpw7bt2for:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp987f45g0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2024-01-21T19:58:51.152 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.159+0000 7f62862de700 1 Processor -- start 2024-01-21T19:58:51.153 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.159+0000 7f62862de700 1 -- start start 2024-01-21T19:58:51.153 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.160+0000 7f62862de700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62800fab80 0x7f62800fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:51.153 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.160+0000 7f62862de700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62800fb490 con 0x7f62800fab80 2024-01-21T19:58:51.153 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.160+0000 7f627ffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62800fab80 0x7f62800fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:51.153 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.160+0000 7f627ffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62800fab80 0x7f62800fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47928/0 (socket says 172.21.15.53:47928) 2024-01-21T19:58:51.153 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.160+0000 7f627ffff700 1 -- 172.21.15.53:0/548422253 learned_addr learned my addr 172.21.15.53:0/548422253 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:58:51.153 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.160+0000 7f627ffff700 1 -- 172.21.15.53:0/548422253 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62800fb5d0 con 0x7f62800fab80 2024-01-21T19:58:51.153 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.160+0000 7f627ffff700 1 --2- 172.21.15.53:0/548422253 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62800fab80 0x7f62800fafa0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f627400b5f0 tx=0x7f6274006ee0).ready entity=mon.0 client_cookie=24aec7a80ab4eea server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:51.156 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.161+0000 7f627effd700 1 -- 172.21.15.53:0/548422253 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f627400d600 con 0x7f62800fab80 2024-01-21T19:58:51.156 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.161+0000 7f627effd700 1 -- 172.21.15.53:0/548422253 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f627400dbd0 con 0x7f62800fab80 2024-01-21T19:58:51.156 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.161+0000 7f627effd700 1 -- 172.21.15.53:0/548422253 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6274011c00 con 0x7f62800fab80 2024-01-21T19:58:51.156 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.161+0000 7f62862de700 1 -- 172.21.15.53:0/548422253 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62800fab80 msgr2=0x7f62800fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:51.156 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.161+0000 7f62862de700 1 --2- 172.21.15.53:0/548422253 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62800fab80 0x7f62800fafa0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f627400b5f0 tx=0x7f6274006ee0).stop 2024-01-21T19:58:51.156 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.161+0000 7f62862de700 1 -- 172.21.15.53:0/548422253 shutdown_connections 2024-01-21T19:58:51.156 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.161+0000 7f62862de700 1 --2- 172.21.15.53:0/548422253 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62800fab80 0x7f62800fafa0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:51.156 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.161+0000 7f62862de700 1 -- 172.21.15.53:0/548422253 >> 172.21.15.53:0/548422253 conn(0x7f62800f63c0 msgr2=0x7f62800f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:58:51.156 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.161+0000 7f62862de700 1 -- 172.21.15.53:0/548422253 shutdown_connections 2024-01-21T19:58:51.156 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.162+0000 7f62862de700 1 -- 172.21.15.53:0/548422253 wait complete. 2024-01-21T19:58:51.156 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.162+0000 7f62862de700 1 Processor -- start 2024-01-21T19:58:51.156 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.162+0000 7f62862de700 1 -- start start 2024-01-21T19:58:51.157 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.163+0000 7f62862de700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62800fab80 0x7f628010cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:51.157 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.163+0000 7f62862de700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628010d0d0 con 0x7f62800fab80 2024-01-21T19:58:51.157 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.163+0000 7f627ffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62800fab80 0x7f628010cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:51.157 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.163+0000 7f627ffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62800fab80 0x7f628010cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47934/0 (socket says 172.21.15.53:47934) 2024-01-21T19:58:51.157 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.163+0000 7f627ffff700 1 -- 172.21.15.53:0/3480937443 learned_addr learned my addr 172.21.15.53:0/3480937443 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:58:51.157 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.163+0000 7f627ffff700 1 -- 172.21.15.53:0/3480937443 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f627400b200 con 0x7f62800fab80 2024-01-21T19:58:51.157 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.163+0000 7f627ffff700 1 --2- 172.21.15.53:0/3480937443 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62800fab80 0x7f628010cbe0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f627400b1d0 tx=0x7f6274006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:51.157 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.163+0000 7f627cff9700 1 -- 172.21.15.53:0/3480937443 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f627401b410 con 0x7f62800fab80 2024-01-21T19:58:51.157 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.163+0000 7f627cff9700 1 -- 172.21.15.53:0/3480937443 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f627401b9e0 con 0x7f62800fab80 2024-01-21T19:58:51.157 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.164+0000 7f62862de700 1 -- 172.21.15.53:0/3480937443 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f628010d2d0 con 0x7f62800fab80 2024-01-21T19:58:51.157 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.164+0000 7f627cff9700 1 -- 172.21.15.53:0/3480937443 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f627401a840 con 0x7f62800fab80 2024-01-21T19:58:51.158 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.164+0000 7f62862de700 1 -- 172.21.15.53:0/3480937443 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f628010d6e0 con 0x7f62800fab80 2024-01-21T19:58:51.158 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.164+0000 7f627cff9700 1 -- 172.21.15.53:0/3480937443 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f627400f070 con 0x7f62800fab80 2024-01-21T19:58:51.158 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.164+0000 7f627cff9700 1 --2- 172.21.15.53:0/3480937443 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7f626c02fd90 0x7f626c032250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:51.158 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.164+0000 7f627cff9700 1 -- 172.21.15.53:0/3480937443 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f6274041200 con 0x7f62800fab80 2024-01-21T19:58:51.158 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.165+0000 7f627f7fe700 1 --2- 172.21.15.53:0/3480937443 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7f626c02fd90 0x7f626c032250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:51.159 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.165+0000 7f627f7fe700 1 --2- 172.21.15.53:0/3480937443 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7f626c02fd90 0x7f626c032250 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f62700096a0 tx=0x7f6270006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:51.159 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.165+0000 7f62862de700 1 -- 172.21.15.53:0/3480937443 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6280059100 con 0x7f62800fab80 2024-01-21T19:58:51.161 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.169+0000 7f627cff9700 1 -- 172.21.15.53:0/3480937443 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6274016070 con 0x7f62800fab80 2024-01-21T19:58:51.358 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.366+0000 7f62862de700 1 -- 172.21.15.53:0/3480937443 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7f62800fc890 con 0x7f62800fab80 2024-01-21T19:58:51.415 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.424+0000 7f627cff9700 1 -- 172.21.15.53:0/3480937443 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f62740257e0 con 0x7f62800fab80 2024-01-21T19:58:51.417 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.425+0000 7f627cff9700 1 -- 172.21.15.53:0/3480937443 <== mon.0 v2:172.21.15.53:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v9) v1 ==== 89+0+0 (secure 0 0 0) 0x7f6274010970 con 0x7f62800fab80 2024-01-21T19:58:51.419 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.427+0000 7f62862de700 1 -- 172.21.15.53:0/3480937443 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7f626c02fd90 msgr2=0x7f626c032250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:51.419 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.427+0000 7f62862de700 1 --2- 172.21.15.53:0/3480937443 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7f626c02fd90 0x7f626c032250 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f62700096a0 tx=0x7f6270006b40).stop 2024-01-21T19:58:51.419 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.427+0000 7f62862de700 1 -- 172.21.15.53:0/3480937443 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62800fab80 msgr2=0x7f628010cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:51.420 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.427+0000 7f62862de700 1 --2- 172.21.15.53:0/3480937443 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62800fab80 0x7f628010cbe0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f627400b1d0 tx=0x7f6274006ee0).stop 2024-01-21T19:58:51.420 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.427+0000 7f62862de700 1 -- 172.21.15.53:0/3480937443 shutdown_connections 2024-01-21T19:58:51.420 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.427+0000 7f62862de700 1 --2- 172.21.15.53:0/3480937443 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7f626c02fd90 0x7f626c032250 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:51.420 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.427+0000 7f62862de700 1 --2- 172.21.15.53:0/3480937443 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62800fab80 0x7f628010cbe0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:51.420 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.427+0000 7f62862de700 1 -- 172.21.15.53:0/3480937443 >> 172.21.15.53:0/3480937443 conn(0x7f62800f63c0 msgr2=0x7f62800ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:58:51.420 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.427+0000 7f62862de700 1 -- 172.21.15.53:0/3480937443 shutdown_connections 2024-01-21T19:58:51.420 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:51.427+0000 7f62862de700 1 -- 172.21.15.53:0/3480937443 wait complete. 2024-01-21T19:58:51.684 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:51 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:50.235035+0000 mgr.smithi053.fgsmol (mgr.14116) 14 : audit [DBG] from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T19:58:51.684 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:51 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cephadm 2024-01-21T19:58:50.235660+0000 mgr.smithi053.fgsmol (mgr.14116) 15 : cephadm [INF] Saving service crash spec with placement * 2024-01-21T19:58:51.684 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:51 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:51.368383+0000 mon.smithi053 (mon.0) 60 : audit [INF] from='client.? 172.21.15.53:0/3480937443' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-01-21T19:58:51.774 INFO:teuthology.orchestra.run.smithi053.stderr:Deploying prometheus service with default placement... 2024-01-21T19:58:51.774 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/57f6722c-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpw7bt2for:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp987f45g0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2024-01-21T19:58:52.158 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:52.165+0000 7fd2c74e3700 1 Processor -- start 2024-01-21T19:58:52.158 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:52.165+0000 7fd2c74e3700 1 -- start start 2024-01-21T19:58:52.158 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:52.165+0000 7fd2c74e3700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd2c00703d0 0x7fd2c0100d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:52.158 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:52.165+0000 7fd2c74e3700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c00fc730 con 0x7fd2c00703d0 2024-01-21T19:58:52.158 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:52.166+0000 7fd2c527f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd2c00703d0 0x7fd2c0100d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:52.158 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:52.166+0000 7fd2c527f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd2c00703d0 0x7fd2c0100d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47964/0 (socket says 172.21.15.53:47964) 2024-01-21T19:58:52.159 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:52.166+0000 7fd2c527f700 1 -- 172.21.15.53:0/1916554271 learned_addr learned my addr 172.21.15.53:0/1916554271 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:58:52.159 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:52.166+0000 7fd2c527f700 1 -- 172.21.15.53:0/1916554271 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c0101230 con 0x7fd2c00703d0 2024-01-21T19:58:52.159 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:52.166+0000 7fd2c527f700 1 --2- 172.21.15.53:0/1916554271 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd2c00703d0 0x7fd2c0100d40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fd2bc00b5f0 tx=0x7fd2bc006ee0).ready entity=mon.0 client_cookie=f2beb599c21217a6 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:52.159 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:52.166+0000 7fd2affff700 1 -- 172.21.15.53:0/1916554271 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd2bc00d600 con 0x7fd2c00703d0 2024-01-21T19:58:52.159 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:52.166+0000 7fd2affff700 1 -- 172.21.15.53:0/1916554271 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd2bc00dbd0 con 0x7fd2c00703d0 2024-01-21T19:58:52.161 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:52.166+0000 7fd2affff700 1 -- 172.21.15.53:0/1916554271 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd2bc011c00 con 0x7fd2c00703d0 2024-01-21T19:58:52.161 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:52.167+0000 7fd2c74e3700 1 -- 172.21.15.53:0/1916554271 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd2c00703d0 msgr2=0x7fd2c0100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:52.161 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:52.167+0000 7fd2c74e3700 1 --2- 172.21.15.53:0/1916554271 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd2c00703d0 0x7fd2c0100d40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fd2bc00b5f0 tx=0x7fd2bc006ee0).stop 2024-01-21T19:58:52.161 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:52.167+0000 7fd2c74e3700 1 -- 172.21.15.53:0/1916554271 shutdown_connections 2024-01-21T19:58:52.161 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:52.167+0000 7fd2c74e3700 1 --2- 172.21.15.53:0/1916554271 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd2c00703d0 0x7fd2c0100d40 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:52.161 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:52.167+0000 7fd2c74e3700 1 -- 172.21.15.53:0/1916554271 >> 172.21.15.53:0/1916554271 conn(0x7fd2c00f63c0 msgr2=0x7fd2c00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:58:52.161 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:52.167+0000 7fd2c74e3700 1 -- 172.21.15.53:0/1916554271 shutdown_connections 2024-01-21T19:58:52.161 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:52.168+0000 7fd2c74e3700 1 -- 172.21.15.53:0/1916554271 wait complete. 2024-01-21T19:58:52.161 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:52.168+0000 7fd2c74e3700 1 Processor -- start 2024-01-21T19:58:52.161 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:52.168+0000 7fd2c74e3700 1 -- start start 2024-01-21T19:58:52.161 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:52.168+0000 7fd2c74e3700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd2c00703d0 0x7fd2c010cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:52.162 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:52.168+0000 7fd2c74e3700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c010d0a0 con 0x7fd2c00703d0 2024-01-21T19:58:52.162 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:52.169+0000 7fd2c527f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd2c00703d0 0x7fd2c010cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:52.162 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:52.169+0000 7fd2c527f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd2c00703d0 0x7fd2c010cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47974/0 (socket says 172.21.15.53:47974) 2024-01-21T19:58:52.162 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:52.169+0000 7fd2c527f700 1 -- 172.21.15.53:0/3115406709 learned_addr learned my addr 172.21.15.53:0/3115406709 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:58:52.162 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:52.169+0000 7fd2c527f700 1 -- 172.21.15.53:0/3115406709 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2bc00b200 con 0x7fd2c00703d0 2024-01-21T19:58:52.162 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:52.169+0000 7fd2c527f700 1 --2- 172.21.15.53:0/3115406709 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd2c00703d0 0x7fd2c010cbb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fd2bc00b190 tx=0x7fd2bc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:52.162 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:52.169+0000 7fd2adffb700 1 -- 172.21.15.53:0/3115406709 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd2bc01b410 con 0x7fd2c00703d0 2024-01-21T19:58:52.162 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:52.169+0000 7fd2c74e3700 1 -- 172.21.15.53:0/3115406709 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2c010d2a0 con 0x7fd2c00703d0 2024-01-21T19:58:52.162 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:52.169+0000 7fd2adffb700 1 -- 172.21.15.53:0/3115406709 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd2bc01b9e0 con 0x7fd2c00703d0 2024-01-21T19:58:52.163 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:52.169+0000 7fd2adffb700 1 -- 172.21.15.53:0/3115406709 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd2bc01a880 con 0x7fd2c00703d0 2024-01-21T19:58:52.163 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:52.170+0000 7fd2c74e3700 1 -- 172.21.15.53:0/3115406709 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2c010d6b0 con 0x7fd2c00703d0 2024-01-21T19:58:52.163 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:52.170+0000 7fd2c74e3700 1 -- 172.21.15.53:0/3115406709 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2c0045ad0 con 0x7fd2c00703d0 2024-01-21T19:58:52.163 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:52.171+0000 7fd2adffb700 1 -- 172.21.15.53:0/3115406709 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7fd2bc00f070 con 0x7fd2c00703d0 2024-01-21T19:58:52.164 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:52.171+0000 7fd2adffb700 1 --2- 172.21.15.53:0/3115406709 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7fd2a802fdb0 0x7fd2a8032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:52.164 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:52.171+0000 7fd2adffb700 1 -- 172.21.15.53:0/3115406709 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fd2bc041e80 con 0x7fd2c00703d0 2024-01-21T19:58:52.164 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:52.171+0000 7fd2c4a7e700 1 -- 172.21.15.53:0/3115406709 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7fd2a802fdb0 msgr2=0x7fd2a8032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/55197364 2024-01-21T19:58:52.164 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:52.171+0000 7fd2c4a7e700 1 --2- 172.21.15.53:0/3115406709 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7fd2a802fdb0 0x7fd2a8032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-21T19:58:52.171 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:52.179+0000 7fd2adffb700 1 -- 172.21.15.53:0/3115406709 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd2bc02d9d0 con 0x7fd2c00703d0 2024-01-21T19:58:52.358 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:52.366+0000 7fd2c74e3700 1 -- 172.21.15.53:0/3115406709 --> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fd2c010dff0 con 0x7fd2a802fdb0 2024-01-21T19:58:52.363 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:52.372+0000 7fd2c4a7e700 1 -- 172.21.15.53:0/3115406709 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7fd2a802fdb0 msgr2=0x7fd2a8032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/55197364 2024-01-21T19:58:52.363 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:52.372+0000 7fd2c4a7e700 1 --2- 172.21.15.53:0/3115406709 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7fd2a802fdb0 0x7fd2a8032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-21T19:58:52.684 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:52 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:51.426635+0000 mon.smithi053 (mon.0) 61 : audit [INF] from='client.? 172.21.15.53:0/3480937443' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-01-21T19:58:52.684 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:52 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:58:51.426690+0000 mon.smithi053 (mon.0) 62 : cluster [DBG] mgrmap e9: smithi053.fgsmol(active, since 17s) 2024-01-21T19:58:52.765 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:52.773+0000 7fd2c4a7e700 1 -- 172.21.15.53:0/3115406709 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7fd2a802fdb0 msgr2=0x7fd2a8032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/55197364 2024-01-21T19:58:52.765 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:52.773+0000 7fd2c4a7e700 1 --2- 172.21.15.53:0/3115406709 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7fd2a802fdb0 0x7fd2a8032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-01-21T19:58:53.566 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:53.574+0000 7fd2c4a7e700 1 -- 172.21.15.53:0/3115406709 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7fd2a802fdb0 msgr2=0x7fd2a8032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/55197364 2024-01-21T19:58:53.566 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:53.574+0000 7fd2c4a7e700 1 --2- 172.21.15.53:0/3115406709 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7fd2a802fdb0 0x7fd2a8032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-01-21T19:58:55.167 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:55.175+0000 7fd2c4a7e700 1 -- 172.21.15.53:0/3115406709 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7fd2a802fdb0 msgr2=0x7fd2a8032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/55197364 2024-01-21T19:58:55.167 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:55.175+0000 7fd2c4a7e700 1 --2- 172.21.15.53:0/3115406709 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7fd2a802fdb0 0x7fd2a8032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-01-21T19:58:57.416 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:57.422+0000 7fd2adffb700 1 -- 172.21.15.53:0/3115406709 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 10) v1 ==== 35984+0+0 (secure 0 0 0) 0x7fd2bc01b570 con 0x7fd2c00703d0 2024-01-21T19:58:57.416 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:57.422+0000 7fd2adffb700 1 -- 172.21.15.53:0/3115406709 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7fd2a802fdb0 msgr2=0x7fd2a8032270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-21T19:58:57.416 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:57.422+0000 7fd2adffb700 1 --2- 172.21.15.53:0/3115406709 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7fd2a802fdb0 0x7fd2a8032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T19:58:57.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:57 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:58:56.594153+0000 mon.smithi053 (mon.0) 63 : cluster [INF] Active manager daemon smithi053.fgsmol restarted 2024-01-21T19:58:57.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:57 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:58:56.594532+0000 mon.smithi053 (mon.0) 64 : cluster [INF] Activating manager daemon smithi053.fgsmol 2024-01-21T19:58:57.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:57 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:58:56.596033+0000 mon.smithi053 (mon.0) 65 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-01-21T19:58:57.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:57 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:58:57.424604+0000 mon.smithi053 (mon.0) 66 : cluster [DBG] mgrmap e10: smithi053.fgsmol(active, starting, since 0.830152s) 2024-01-21T19:58:57.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:57 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:57.426248+0000 mon.smithi053 (mon.0) 67 : audit [DBG] from='mgr.14144 172.21.15.53:0/902443509' entity='mgr.smithi053.fgsmol' cmd=[{"prefix": "mon metadata", "id": "smithi053"}]: dispatch 2024-01-21T19:58:57.935 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:57 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:57.427093+0000 mon.smithi053 (mon.0) 68 : audit [DBG] from='mgr.14144 172.21.15.53:0/902443509' entity='mgr.smithi053.fgsmol' cmd=[{"prefix": "mgr metadata", "who": "smithi053.fgsmol", "id": "smithi053.fgsmol"}]: dispatch 2024-01-21T19:58:57.935 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:57 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:57.428001+0000 mon.smithi053 (mon.0) 69 : audit [DBG] from='mgr.14144 172.21.15.53:0/902443509' entity='mgr.smithi053.fgsmol' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-21T19:58:57.935 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:57 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:57.428321+0000 mon.smithi053 (mon.0) 70 : audit [DBG] from='mgr.14144 172.21.15.53:0/902443509' entity='mgr.smithi053.fgsmol' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-21T19:58:57.935 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:57 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:57.428627+0000 mon.smithi053 (mon.0) 71 : audit [DBG] from='mgr.14144 172.21.15.53:0/902443509' entity='mgr.smithi053.fgsmol' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-21T19:58:57.935 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:57 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:58:57.443186+0000 mon.smithi053 (mon.0) 72 : cluster [INF] Manager daemon smithi053.fgsmol is now available 2024-01-21T19:58:57.935 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:57 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:57.558447+0000 mon.smithi053 (mon.0) 73 : audit [DBG] from='mgr.14144 172.21.15.53:0/902443509' entity='mgr.smithi053.fgsmol' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T19:58:57.935 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:57 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:57.559710+0000 mon.smithi053 (mon.0) 74 : audit [DBG] from='mgr.14144 172.21.15.53:0/902443509' entity='mgr.smithi053.fgsmol' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T19:58:57.935 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:57 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:57.576659+0000 mon.smithi053 (mon.0) 75 : audit [INF] from='mgr.14144 172.21.15.53:0/902443509' entity='mgr.smithi053.fgsmol' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fgsmol/mirror_snapshot_schedule"}]: dispatch 2024-01-21T19:58:57.935 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:57 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:57.584509+0000 mon.smithi053 (mon.0) 76 : audit [INF] from='mgr.14144 172.21.15.53:0/902443509' entity='mgr.smithi053.fgsmol' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fgsmol/trash_purge_schedule"}]: dispatch 2024-01-21T19:58:58.416 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:58.423+0000 7fd2adffb700 1 -- 172.21.15.53:0/3115406709 <== mon.0 v2:172.21.15.53:3300/0 8 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7fd2bc01a9e0 con 0x7fd2c00703d0 2024-01-21T19:58:58.417 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:58.424+0000 7fd2adffb700 1 --2- 172.21.15.53:0/3115406709 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7fd2a8033c20 0x7fd2a8036010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:58.417 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:58.424+0000 7fd2adffb700 1 -- 172.21.15.53:0/3115406709 --> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fd2c010dff0 con 0x7fd2a8033c20 2024-01-21T19:58:58.417 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:58.424+0000 7fd2c4a7e700 1 --2- 172.21.15.53:0/3115406709 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7fd2a8033c20 0x7fd2a8036010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:58.417 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:58.425+0000 7fd2c4a7e700 1 --2- 172.21.15.53:0/3115406709 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7fd2a8033c20 0x7fd2a8036010 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fd2b00036d0 tx=0x7fd2b0006cb0).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:58.421 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:58.429+0000 7fd2adffb700 1 -- 172.21.15.53:0/3115406709 <== mgr.14144 v2:172.21.15.53:6800/3045083722 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7fd2c010dff0 con 0x7fd2a8033c20 2024-01-21T19:58:58.421 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-01-21T19:58:58.424 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:58.431+0000 7fd2c74e3700 1 -- 172.21.15.53:0/3115406709 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7fd2a8033c20 msgr2=0x7fd2a8036010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:58.424 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:58.431+0000 7fd2c74e3700 1 --2- 172.21.15.53:0/3115406709 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7fd2a8033c20 0x7fd2a8036010 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fd2b00036d0 tx=0x7fd2b0006cb0).stop 2024-01-21T19:58:58.424 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:58.431+0000 7fd2c74e3700 1 -- 172.21.15.53:0/3115406709 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd2c00703d0 msgr2=0x7fd2c010cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:58.424 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:58.431+0000 7fd2c74e3700 1 --2- 172.21.15.53:0/3115406709 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd2c00703d0 0x7fd2c010cbb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fd2bc00b190 tx=0x7fd2bc006ee0).stop 2024-01-21T19:58:58.425 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:58.431+0000 7fd2c74e3700 1 -- 172.21.15.53:0/3115406709 shutdown_connections 2024-01-21T19:58:58.425 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:58.431+0000 7fd2c74e3700 1 --2- 172.21.15.53:0/3115406709 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7fd2a8033c20 0x7fd2a8036010 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:58.425 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:58.431+0000 7fd2c74e3700 1 --2- 172.21.15.53:0/3115406709 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd2c00703d0 0x7fd2c010cbb0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:58.425 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:58.431+0000 7fd2c74e3700 1 --2- 172.21.15.53:0/3115406709 >> [v2:172.21.15.53:6800/55197364,v1:172.21.15.53:6801/55197364] conn(0x7fd2a802fdb0 0x7fd2a8032270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T19:58:58.425 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:58.431+0000 7fd2c74e3700 1 -- 172.21.15.53:0/3115406709 >> 172.21.15.53:0/3115406709 conn(0x7fd2c00f63c0 msgr2=0x7fd2c00f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:58:58.425 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:58.431+0000 7fd2c74e3700 1 -- 172.21.15.53:0/3115406709 shutdown_connections 2024-01-21T19:58:58.425 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:58.431+0000 7fd2c74e3700 1 -- 172.21.15.53:0/3115406709 wait complete. 2024-01-21T19:58:58.740 INFO:teuthology.orchestra.run.smithi053.stderr:Deploying grafana service with default placement... 2024-01-21T19:58:58.740 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/57f6722c-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpw7bt2for:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp987f45g0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2024-01-21T19:58:59.104 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.106+0000 7fb9cb150700 1 Processor -- start 2024-01-21T19:58:59.104 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.106+0000 7fb9cb150700 1 -- start start 2024-01-21T19:58:59.105 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.107+0000 7fb9cb150700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb9c40703b0 0x7fb9c4100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:59.105 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.107+0000 7fb9cb150700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c40fc750 con 0x7fb9c40703b0 2024-01-21T19:58:59.105 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.107+0000 7fb9c8eec700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb9c40703b0 0x7fb9c4100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:59.105 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.107+0000 7fb9c8eec700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb9c40703b0 0x7fb9c4100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:37362/0 (socket says 172.21.15.53:37362) 2024-01-21T19:58:59.105 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.107+0000 7fb9c8eec700 1 -- 172.21.15.53:0/2192009007 learned_addr learned my addr 172.21.15.53:0/2192009007 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:58:59.105 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.107+0000 7fb9c8eec700 1 -- 172.21.15.53:0/2192009007 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c4101250 con 0x7fb9c40703b0 2024-01-21T19:58:59.105 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.107+0000 7fb9c8eec700 1 --2- 172.21.15.53:0/2192009007 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb9c40703b0 0x7fb9c4100d60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fb9b800b5f0 tx=0x7fb9b8006ee0).ready entity=mon.0 client_cookie=c9c15028bde944a5 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:59.105 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.107+0000 7fb9c37fe700 1 -- 172.21.15.53:0/2192009007 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9b800d600 con 0x7fb9c40703b0 2024-01-21T19:58:59.105 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.107+0000 7fb9c37fe700 1 -- 172.21.15.53:0/2192009007 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb9b800dbd0 con 0x7fb9c40703b0 2024-01-21T19:58:59.105 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.107+0000 7fb9c37fe700 1 -- 172.21.15.53:0/2192009007 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9b8011c00 con 0x7fb9c40703b0 2024-01-21T19:58:59.106 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.108+0000 7fb9cb150700 1 -- 172.21.15.53:0/2192009007 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb9c40703b0 msgr2=0x7fb9c4100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:59.106 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.108+0000 7fb9cb150700 1 --2- 172.21.15.53:0/2192009007 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb9c40703b0 0x7fb9c4100d60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fb9b800b5f0 tx=0x7fb9b8006ee0).stop 2024-01-21T19:58:59.106 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.108+0000 7fb9cb150700 1 -- 172.21.15.53:0/2192009007 shutdown_connections 2024-01-21T19:58:59.106 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.108+0000 7fb9cb150700 1 --2- 172.21.15.53:0/2192009007 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb9c40703b0 0x7fb9c4100d60 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:59.106 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.108+0000 7fb9cb150700 1 -- 172.21.15.53:0/2192009007 >> 172.21.15.53:0/2192009007 conn(0x7fb9c40f63c0 msgr2=0x7fb9c40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:58:59.106 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.108+0000 7fb9cb150700 1 -- 172.21.15.53:0/2192009007 shutdown_connections 2024-01-21T19:58:59.106 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.108+0000 7fb9cb150700 1 -- 172.21.15.53:0/2192009007 wait complete. 2024-01-21T19:58:59.107 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.109+0000 7fb9cb150700 1 Processor -- start 2024-01-21T19:58:59.107 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.109+0000 7fb9cb150700 1 -- start start 2024-01-21T19:58:59.107 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.109+0000 7fb9cb150700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb9c40703b0 0x7fb9c410ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:59.107 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.109+0000 7fb9cb150700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c410d1e0 con 0x7fb9c40703b0 2024-01-21T19:58:59.107 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.109+0000 7fb9c8eec700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb9c40703b0 0x7fb9c410ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:59.107 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.109+0000 7fb9c8eec700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb9c40703b0 0x7fb9c410ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:37374/0 (socket says 172.21.15.53:37374) 2024-01-21T19:58:59.107 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.109+0000 7fb9c8eec700 1 -- 172.21.15.53:0/3028328472 learned_addr learned my addr 172.21.15.53:0/3028328472 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:58:59.107 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.110+0000 7fb9c8eec700 1 -- 172.21.15.53:0/3028328472 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9b800b200 con 0x7fb9c40703b0 2024-01-21T19:58:59.107 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.110+0000 7fb9c8eec700 1 --2- 172.21.15.53:0/3028328472 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb9c40703b0 0x7fb9c410ccf0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fb9b800e040 tx=0x7fb9b8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:59.108 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.110+0000 7fb9c17fa700 1 -- 172.21.15.53:0/3028328472 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9b800d9f0 con 0x7fb9c40703b0 2024-01-21T19:58:59.108 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.110+0000 7fb9cb150700 1 -- 172.21.15.53:0/3028328472 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9c410d3e0 con 0x7fb9c40703b0 2024-01-21T19:58:59.108 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.110+0000 7fb9c17fa700 1 -- 172.21.15.53:0/3028328472 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb9b801a9f0 con 0x7fb9c40703b0 2024-01-21T19:58:59.108 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.110+0000 7fb9c17fa700 1 -- 172.21.15.53:0/3028328472 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9b801b8c0 con 0x7fb9c40703b0 2024-01-21T19:58:59.108 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.110+0000 7fb9cb150700 1 -- 172.21.15.53:0/3028328472 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9c410d7f0 con 0x7fb9c40703b0 2024-01-21T19:58:59.108 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.111+0000 7fb9c17fa700 1 -- 172.21.15.53:0/3028328472 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7fb9b800f070 con 0x7fb9c40703b0 2024-01-21T19:58:59.108 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.111+0000 7fb9cb150700 1 -- 172.21.15.53:0/3028328472 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9c4045ad0 con 0x7fb9c40703b0 2024-01-21T19:58:59.109 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.111+0000 7fb9c17fa700 1 --2- 172.21.15.53:0/3028328472 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7fb9a802fd70 0x7fb9a8032230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:59.109 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.111+0000 7fb9c17fa700 1 -- 172.21.15.53:0/3028328472 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fb9b8041260 con 0x7fb9c40703b0 2024-01-21T19:58:59.109 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.111+0000 7fb9c3fff700 1 --2- 172.21.15.53:0/3028328472 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7fb9a802fd70 0x7fb9a8032230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:59.109 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.111+0000 7fb9c3fff700 1 --2- 172.21.15.53:0/3028328472 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7fb9a802fd70 0x7fb9a8032230 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fb9b00096a0 tx=0x7fb9b0006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:59.109 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.116+0000 7fb9c17fa700 1 -- 172.21.15.53:0/3028328472 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb9b8019b90 con 0x7fb9c40703b0 2024-01-21T19:58:59.295 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.303+0000 7fb9cb150700 1 -- 172.21.15.53:0/3028328472 --> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7fb9c410df20 con 0x7fb9a802fd70 2024-01-21T19:58:59.298 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.306+0000 7fb9c17fa700 1 -- 172.21.15.53:0/3028328472 <== mgr.14144 v2:172.21.15.53:6800/3045083722 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fb9c410df20 con 0x7fb9a802fd70 2024-01-21T19:58:59.298 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-01-21T19:58:59.301 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.308+0000 7fb9cb150700 1 -- 172.21.15.53:0/3028328472 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7fb9a802fd70 msgr2=0x7fb9a8032230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:59.301 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.308+0000 7fb9cb150700 1 --2- 172.21.15.53:0/3028328472 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7fb9a802fd70 0x7fb9a8032230 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fb9b00096a0 tx=0x7fb9b0006b40).stop 2024-01-21T19:58:59.301 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.308+0000 7fb9cb150700 1 -- 172.21.15.53:0/3028328472 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb9c40703b0 msgr2=0x7fb9c410ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:59.301 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.308+0000 7fb9cb150700 1 --2- 172.21.15.53:0/3028328472 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb9c40703b0 0x7fb9c410ccf0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fb9b800e040 tx=0x7fb9b8006ee0).stop 2024-01-21T19:58:59.301 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.308+0000 7fb9cb150700 1 -- 172.21.15.53:0/3028328472 shutdown_connections 2024-01-21T19:58:59.301 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.308+0000 7fb9cb150700 1 --2- 172.21.15.53:0/3028328472 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7fb9a802fd70 0x7fb9a8032230 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:59.301 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.308+0000 7fb9cb150700 1 --2- 172.21.15.53:0/3028328472 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb9c40703b0 0x7fb9c410ccf0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:59.301 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.308+0000 7fb9cb150700 1 -- 172.21.15.53:0/3028328472 >> 172.21.15.53:0/3028328472 conn(0x7fb9c40f63c0 msgr2=0x7fb9c40f7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:58:59.301 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.308+0000 7fb9cb150700 1 -- 172.21.15.53:0/3028328472 shutdown_connections 2024-01-21T19:58:59.301 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.308+0000 7fb9cb150700 1 -- 172.21.15.53:0/3028328472 wait complete. 2024-01-21T19:58:59.604 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:59 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:58:58.427416+0000 mon.smithi053 (mon.0) 77 : cluster [DBG] mgrmap e11: smithi053.fgsmol(active, since 1.83297s) 2024-01-21T19:58:59.604 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:59 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cephadm 2024-01-21T19:58:58.428259+0000 mgr.smithi053.fgsmol (mgr.14144) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2024-01-21T19:58:59.604 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:59 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:58.430393+0000 mon.smithi053 (mon.0) 78 : audit [INF] from='mgr.14144 172.21.15.53:0/902443509' entity='mgr.smithi053.fgsmol' 2024-01-21T19:58:59.604 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:58:59 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:59.307814+0000 mon.smithi053 (mon.0) 79 : audit [INF] from='mgr.14144 172.21.15.53:0/902443509' entity='mgr.smithi053.fgsmol' 2024-01-21T19:58:59.640 INFO:teuthology.orchestra.run.smithi053.stderr:Deploying node-exporter service with default placement... 2024-01-21T19:58:59.640 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/57f6722c-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpw7bt2for:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp987f45g0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2024-01-21T19:58:59.991 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.998+0000 7f2e340e9700 1 Processor -- start 2024-01-21T19:58:59.991 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.998+0000 7f2e340e9700 1 -- start start 2024-01-21T19:58:59.991 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.999+0000 7f2e340e9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e2c0703d0 0x7f2e2c100d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:59.991 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.999+0000 7f2e340e9700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e2c0fc730 con 0x7f2e2c0703d0 2024-01-21T19:58:59.992 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.999+0000 7f2e31e85700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e2c0703d0 0x7f2e2c100d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:59.992 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.999+0000 7f2e31e85700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e2c0703d0 0x7f2e2c100d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:37380/0 (socket says 172.21.15.53:37380) 2024-01-21T19:58:59.993 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.999+0000 7f2e31e85700 1 -- 172.21.15.53:0/3573646213 learned_addr learned my addr 172.21.15.53:0/3573646213 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:58:59.993 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.999+0000 7f2e31e85700 1 -- 172.21.15.53:0/3573646213 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e2c101230 con 0x7f2e2c0703d0 2024-01-21T19:58:59.993 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.999+0000 7f2e31e85700 1 --2- 172.21.15.53:0/3573646213 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e2c0703d0 0x7f2e2c100d40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f2e1c00b5f0 tx=0x7f2e1c006ee0).ready entity=mon.0 client_cookie=c139ad325e4ffd24 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:59.993 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.999+0000 7f2e30e83700 1 -- 172.21.15.53:0/3573646213 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e1c00d600 con 0x7f2e2c0703d0 2024-01-21T19:58:59.993 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.999+0000 7f2e30e83700 1 -- 172.21.15.53:0/3573646213 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f2e1c00dbd0 con 0x7f2e2c0703d0 2024-01-21T19:58:59.993 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:58:59.999+0000 7f2e30e83700 1 -- 172.21.15.53:0/3573646213 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e1c011c00 con 0x7f2e2c0703d0 2024-01-21T19:58:59.993 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.000+0000 7f2e340e9700 1 -- 172.21.15.53:0/3573646213 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e2c0703d0 msgr2=0x7f2e2c100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:58:59.993 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.000+0000 7f2e340e9700 1 --2- 172.21.15.53:0/3573646213 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e2c0703d0 0x7f2e2c100d40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f2e1c00b5f0 tx=0x7f2e1c006ee0).stop 2024-01-21T19:58:59.993 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.000+0000 7f2e340e9700 1 -- 172.21.15.53:0/3573646213 shutdown_connections 2024-01-21T19:58:59.993 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.000+0000 7f2e340e9700 1 --2- 172.21.15.53:0/3573646213 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e2c0703d0 0x7f2e2c100d40 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:58:59.993 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.000+0000 7f2e340e9700 1 -- 172.21.15.53:0/3573646213 >> 172.21.15.53:0/3573646213 conn(0x7f2e2c0f63c0 msgr2=0x7f2e2c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:58:59.994 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.001+0000 7f2e340e9700 1 -- 172.21.15.53:0/3573646213 shutdown_connections 2024-01-21T19:58:59.994 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.001+0000 7f2e340e9700 1 -- 172.21.15.53:0/3573646213 wait complete. 2024-01-21T19:58:59.996 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.002+0000 7f2e340e9700 1 Processor -- start 2024-01-21T19:58:59.996 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.002+0000 7f2e340e9700 1 -- start start 2024-01-21T19:58:59.996 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.002+0000 7f2e340e9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e2c0703d0 0x7f2e2c10cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:59.996 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.002+0000 7f2e340e9700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e2c10d0a0 con 0x7f2e2c0703d0 2024-01-21T19:58:59.996 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.002+0000 7f2e31e85700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e2c0703d0 0x7f2e2c10cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:59.996 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.002+0000 7f2e31e85700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e2c0703d0 0x7f2e2c10cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:37396/0 (socket says 172.21.15.53:37396) 2024-01-21T19:58:59.996 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.002+0000 7f2e31e85700 1 -- 172.21.15.53:0/3974660467 learned_addr learned my addr 172.21.15.53:0/3974660467 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:58:59.996 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.002+0000 7f2e31e85700 1 -- 172.21.15.53:0/3974660467 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e1c00b200 con 0x7f2e2c0703d0 2024-01-21T19:58:59.996 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.002+0000 7f2e31e85700 1 --2- 172.21.15.53:0/3974660467 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e2c0703d0 0x7f2e2c10cbb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f2e1c00b190 tx=0x7f2e1c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:58:59.996 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.003+0000 7f2e1a7fc700 1 -- 172.21.15.53:0/3974660467 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e1c01b410 con 0x7f2e2c0703d0 2024-01-21T19:58:59.996 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.003+0000 7f2e1a7fc700 1 -- 172.21.15.53:0/3974660467 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f2e1c01b9e0 con 0x7f2e2c0703d0 2024-01-21T19:58:59.997 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.003+0000 7f2e340e9700 1 -- 172.21.15.53:0/3974660467 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e2c10d2a0 con 0x7f2e2c0703d0 2024-01-21T19:58:59.997 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.003+0000 7f2e1a7fc700 1 -- 172.21.15.53:0/3974660467 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e1c01a880 con 0x7f2e2c0703d0 2024-01-21T19:58:59.997 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.003+0000 7f2e340e9700 1 -- 172.21.15.53:0/3974660467 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e2c10d6b0 con 0x7f2e2c0703d0 2024-01-21T19:58:59.997 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.003+0000 7f2e1a7fc700 1 -- 172.21.15.53:0/3974660467 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f2e1c00f070 con 0x7f2e2c0703d0 2024-01-21T19:58:59.997 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.004+0000 7f2e1a7fc700 1 --2- 172.21.15.53:0/3974660467 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7f2e1402feb0 0x7f2e14032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:58:59.997 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.004+0000 7f2e1a7fc700 1 -- 172.21.15.53:0/3974660467 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f2e1c041360 con 0x7f2e2c0703d0 2024-01-21T19:58:59.997 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.004+0000 7f2e31684700 1 --2- 172.21.15.53:0/3974660467 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7f2e1402feb0 0x7f2e14032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:58:59.997 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.004+0000 7f2e340e9700 1 -- 172.21.15.53:0/3974660467 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e2c059100 con 0x7f2e2c0703d0 2024-01-21T19:58:59.997 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.004+0000 7f2e31684700 1 --2- 172.21.15.53:0/3974660467 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7f2e1402feb0 0x7f2e14032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f2e280096a0 tx=0x7f2e28006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:00.000 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.008+0000 7f2e1a7fc700 1 -- 172.21.15.53:0/3974660467 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2e1c016070 con 0x7f2e2c0703d0 2024-01-21T19:59:00.188 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.196+0000 7f2e340e9700 1 -- 172.21.15.53:0/3974660467 --> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f2e2c106af0 con 0x7f2e1402feb0 2024-01-21T19:59:00.192 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.200+0000 7f2e1a7fc700 1 -- 172.21.15.53:0/3974660467 <== mgr.14144 v2:172.21.15.53:6800/3045083722 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f2e2c106af0 con 0x7f2e1402feb0 2024-01-21T19:59:00.192 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-01-21T19:59:00.194 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.202+0000 7f2e340e9700 1 -- 172.21.15.53:0/3974660467 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7f2e1402feb0 msgr2=0x7f2e14032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:00.195 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.202+0000 7f2e340e9700 1 --2- 172.21.15.53:0/3974660467 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7f2e1402feb0 0x7f2e14032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f2e280096a0 tx=0x7f2e28006b40).stop 2024-01-21T19:59:00.195 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.202+0000 7f2e340e9700 1 -- 172.21.15.53:0/3974660467 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e2c0703d0 msgr2=0x7f2e2c10cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:00.195 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.202+0000 7f2e340e9700 1 --2- 172.21.15.53:0/3974660467 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e2c0703d0 0x7f2e2c10cbb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f2e1c00b190 tx=0x7f2e1c006ee0).stop 2024-01-21T19:59:00.195 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.202+0000 7f2e340e9700 1 -- 172.21.15.53:0/3974660467 shutdown_connections 2024-01-21T19:59:00.195 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.202+0000 7f2e340e9700 1 --2- 172.21.15.53:0/3974660467 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7f2e1402feb0 0x7f2e14032370 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:00.195 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.202+0000 7f2e340e9700 1 --2- 172.21.15.53:0/3974660467 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e2c0703d0 0x7f2e2c10cbb0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:00.195 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.202+0000 7f2e340e9700 1 -- 172.21.15.53:0/3974660467 >> 172.21.15.53:0/3974660467 conn(0x7f2e2c0f63c0 msgr2=0x7f2e2c0f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:00.195 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.203+0000 7f2e340e9700 1 -- 172.21.15.53:0/3974660467 shutdown_connections 2024-01-21T19:59:00.195 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.203+0000 7f2e340e9700 1 -- 172.21.15.53:0/3974660467 wait complete. 2024-01-21T19:59:00.491 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:00 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:58:59.305243+0000 mgr.smithi053.fgsmol (mgr.14144) 3 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T19:59:00.491 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:00 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cephadm 2024-01-21T19:58:59.305729+0000 mgr.smithi053.fgsmol (mgr.14144) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2024-01-21T19:59:00.492 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:00 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:58:59.431497+0000 mon.smithi053 (mon.0) 80 : cluster [DBG] mgrmap e12: smithi053.fgsmol(active, since 2s) 2024-01-21T19:59:00.492 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:00 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:59:00.201708+0000 mon.smithi053 (mon.0) 81 : audit [INF] from='mgr.14144 172.21.15.53:0/902443509' entity='mgr.smithi053.fgsmol' 2024-01-21T19:59:00.547 INFO:teuthology.orchestra.run.smithi053.stderr:Deploying alertmanager service with default placement... 2024-01-21T19:59:00.547 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/57f6722c-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpw7bt2for:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp987f45g0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2024-01-21T19:59:00.891 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.897+0000 7f6ba9e96700 1 Processor -- start 2024-01-21T19:59:00.891 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.897+0000 7f6ba9e96700 1 -- start start 2024-01-21T19:59:00.891 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.897+0000 7f6ba9e96700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ba40fa9d0 0x7f6ba40fadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:00.891 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.897+0000 7f6ba9e96700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba40fb2e0 con 0x7f6ba40fa9d0 2024-01-21T19:59:00.891 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.897+0000 7f6ba8e94700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ba40fa9d0 0x7f6ba40fadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:00.892 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.897+0000 7f6ba8e94700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ba40fa9d0 0x7f6ba40fadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:37404/0 (socket says 172.21.15.53:37404) 2024-01-21T19:59:00.892 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.897+0000 7f6ba8e94700 1 -- 172.21.15.53:0/2175854708 learned_addr learned my addr 172.21.15.53:0/2175854708 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:59:00.892 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.898+0000 7f6ba8e94700 1 -- 172.21.15.53:0/2175854708 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ba40fb420 con 0x7f6ba40fa9d0 2024-01-21T19:59:00.892 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.898+0000 7f6ba8e94700 1 --2- 172.21.15.53:0/2175854708 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ba40fa9d0 0x7f6ba40fadf0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f6b9800b870 tx=0x7f6b98006f70).ready entity=mon.0 client_cookie=26ad2355c722673 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:00.892 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.898+0000 7f6ba37fe700 1 -- 172.21.15.53:0/2175854708 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b9800da20 con 0x7f6ba40fa9d0 2024-01-21T19:59:00.892 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.898+0000 7f6ba37fe700 1 -- 172.21.15.53:0/2175854708 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f6b98010470 con 0x7f6ba40fa9d0 2024-01-21T19:59:00.892 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.898+0000 7f6ba37fe700 1 -- 172.21.15.53:0/2175854708 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b9800f5b0 con 0x7f6ba40fa9d0 2024-01-21T19:59:00.892 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.898+0000 7f6ba9e96700 1 -- 172.21.15.53:0/2175854708 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ba40fa9d0 msgr2=0x7f6ba40fadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:00.892 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.898+0000 7f6ba9e96700 1 --2- 172.21.15.53:0/2175854708 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ba40fa9d0 0x7f6ba40fadf0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f6b9800b870 tx=0x7f6b98006f70).stop 2024-01-21T19:59:00.892 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.898+0000 7f6ba9e96700 1 -- 172.21.15.53:0/2175854708 shutdown_connections 2024-01-21T19:59:00.892 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.898+0000 7f6ba9e96700 1 --2- 172.21.15.53:0/2175854708 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ba40fa9d0 0x7f6ba40fadf0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:00.892 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.898+0000 7f6ba9e96700 1 -- 172.21.15.53:0/2175854708 >> 172.21.15.53:0/2175854708 conn(0x7f6ba40f6210 msgr2=0x7f6ba40f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:00.892 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.898+0000 7f6ba9e96700 1 -- 172.21.15.53:0/2175854708 shutdown_connections 2024-01-21T19:59:00.893 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.899+0000 7f6ba9e96700 1 -- 172.21.15.53:0/2175854708 wait complete. 2024-01-21T19:59:00.893 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.899+0000 7f6ba9e96700 1 Processor -- start 2024-01-21T19:59:00.893 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.899+0000 7f6ba9e96700 1 -- start start 2024-01-21T19:59:00.895 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.900+0000 7f6ba9e96700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ba40fa9d0 0x7f6ba4108570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:00.895 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.900+0000 7f6ba9e96700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba4108a60 con 0x7f6ba40fa9d0 2024-01-21T19:59:00.895 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.900+0000 7f6ba8e94700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ba40fa9d0 0x7f6ba4108570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:00.895 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.900+0000 7f6ba8e94700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ba40fa9d0 0x7f6ba4108570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:37418/0 (socket says 172.21.15.53:37418) 2024-01-21T19:59:00.895 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.900+0000 7f6ba8e94700 1 -- 172.21.15.53:0/856164562 learned_addr learned my addr 172.21.15.53:0/856164562 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:59:00.895 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.900+0000 7f6ba8e94700 1 -- 172.21.15.53:0/856164562 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b9800b480 con 0x7f6ba40fa9d0 2024-01-21T19:59:00.895 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.900+0000 7f6ba8e94700 1 --2- 172.21.15.53:0/856164562 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ba40fa9d0 0x7f6ba4108570 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f6b98006a40 tx=0x7f6b98006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:00.895 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.900+0000 7f6ba17fa700 1 -- 172.21.15.53:0/856164562 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b9800f3d0 con 0x7f6ba40fa9d0 2024-01-21T19:59:00.895 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.900+0000 7f6ba17fa700 1 -- 172.21.15.53:0/856164562 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f6b9800fef0 con 0x7f6ba40fa9d0 2024-01-21T19:59:00.895 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.900+0000 7f6ba9e96700 1 -- 172.21.15.53:0/856164562 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ba4108c60 con 0x7f6ba40fa9d0 2024-01-21T19:59:00.895 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.900+0000 7f6ba17fa700 1 -- 172.21.15.53:0/856164562 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b98019d40 con 0x7f6ba40fa9d0 2024-01-21T19:59:00.896 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.900+0000 7f6ba9e96700 1 -- 172.21.15.53:0/856164562 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ba4109070 con 0x7f6ba40fa9d0 2024-01-21T19:59:00.896 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.901+0000 7f6ba17fa700 1 -- 172.21.15.53:0/856164562 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f6b9802b060 con 0x7f6ba40fa9d0 2024-01-21T19:59:00.896 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.901+0000 7f6ba17fa700 1 --2- 172.21.15.53:0/856164562 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7f6b8802fe60 0x7f6b88032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:00.896 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.901+0000 7f6ba17fa700 1 -- 172.21.15.53:0/856164562 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f6b98044230 con 0x7f6ba40fa9d0 2024-01-21T19:59:00.896 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.901+0000 7f6ba9e96700 1 -- 172.21.15.53:0/856164562 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ba4045ad0 con 0x7f6ba40fa9d0 2024-01-21T19:59:00.896 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.901+0000 7f6ba3fff700 1 --2- 172.21.15.53:0/856164562 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7f6b8802fe60 0x7f6b88032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:00.896 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.902+0000 7f6ba3fff700 1 --2- 172.21.15.53:0/856164562 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7f6b8802fe60 0x7f6b88032320 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f6b900096a0 tx=0x7f6b90006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:00.897 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:00.906+0000 7f6ba17fa700 1 -- 172.21.15.53:0/856164562 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6b98018ba0 con 0x7f6ba40fa9d0 2024-01-21T19:59:01.085 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.093+0000 7f6ba9e96700 1 -- 172.21.15.53:0/856164562 --> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f6ba4109950 con 0x7f6b8802fe60 2024-01-21T19:59:01.088 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.096+0000 7f6ba17fa700 1 -- 172.21.15.53:0/856164562 <== mgr.14144 v2:172.21.15.53:6800/3045083722 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f6ba4109950 con 0x7f6b8802fe60 2024-01-21T19:59:01.088 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-01-21T19:59:01.090 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.098+0000 7f6ba9e96700 1 -- 172.21.15.53:0/856164562 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7f6b8802fe60 msgr2=0x7f6b88032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:01.091 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.098+0000 7f6ba9e96700 1 --2- 172.21.15.53:0/856164562 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7f6b8802fe60 0x7f6b88032320 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f6b900096a0 tx=0x7f6b90006b40).stop 2024-01-21T19:59:01.091 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.098+0000 7f6ba9e96700 1 -- 172.21.15.53:0/856164562 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ba40fa9d0 msgr2=0x7f6ba4108570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:01.091 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.098+0000 7f6ba9e96700 1 --2- 172.21.15.53:0/856164562 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ba40fa9d0 0x7f6ba4108570 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f6b98006a40 tx=0x7f6b98006f70).stop 2024-01-21T19:59:01.091 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.098+0000 7f6ba9e96700 1 -- 172.21.15.53:0/856164562 shutdown_connections 2024-01-21T19:59:01.091 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.098+0000 7f6ba9e96700 1 --2- 172.21.15.53:0/856164562 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7f6b8802fe60 0x7f6b88032320 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:01.091 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.098+0000 7f6ba9e96700 1 --2- 172.21.15.53:0/856164562 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ba40fa9d0 0x7f6ba4108570 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:01.091 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.098+0000 7f6ba9e96700 1 -- 172.21.15.53:0/856164562 >> 172.21.15.53:0/856164562 conn(0x7f6ba40f6210 msgr2=0x7f6ba40ff760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:01.091 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.098+0000 7f6ba9e96700 1 -- 172.21.15.53:0/856164562 shutdown_connections 2024-01-21T19:59:01.091 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.098+0000 7f6ba9e96700 1 -- 172.21.15.53:0/856164562 wait complete. 2024-01-21T19:59:01.422 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/57f6722c-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpw7bt2for:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp987f45g0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2024-01-21T19:59:01.684 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:01 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:59:00.198745+0000 mgr.smithi053.fgsmol (mgr.14144) 5 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T19:59:01.684 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:01 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cephadm 2024-01-21T19:59:00.199413+0000 mgr.smithi053.fgsmol (mgr.14144) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2024-01-21T19:59:01.684 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:01 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:59:01.097726+0000 mon.smithi053 (mon.0) 82 : audit [INF] from='mgr.14144 172.21.15.53:0/902443509' entity='mgr.smithi053.fgsmol' 2024-01-21T19:59:01.781 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.788+0000 7fd121da0700 1 Processor -- start 2024-01-21T19:59:01.781 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.788+0000 7fd121da0700 1 -- start start 2024-01-21T19:59:01.781 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.788+0000 7fd121da0700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd11c0724b0 0x7fd11c070a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:01.781 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.788+0000 7fd121da0700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd11c0728d0 con 0x7fd11c0724b0 2024-01-21T19:59:01.781 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.788+0000 7fd11b7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd11c0724b0 0x7fd11c070a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:01.782 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.789+0000 7fd11b7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd11c0724b0 0x7fd11c070a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:37434/0 (socket says 172.21.15.53:37434) 2024-01-21T19:59:01.782 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.789+0000 7fd11b7fe700 1 -- 172.21.15.53:0/1920929903 learned_addr learned my addr 172.21.15.53:0/1920929903 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:59:01.782 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.789+0000 7fd11b7fe700 1 -- 172.21.15.53:0/1920929903 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd11c070f40 con 0x7fd11c0724b0 2024-01-21T19:59:01.782 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.789+0000 7fd11b7fe700 1 --2- 172.21.15.53:0/1920929903 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd11c0724b0 0x7fd11c070a50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fd10400b5f0 tx=0x7fd104006ee0).ready entity=mon.0 client_cookie=93453cda0029879f server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:01.782 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.789+0000 7fd11a7fc700 1 -- 172.21.15.53:0/1920929903 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd10400d600 con 0x7fd11c0724b0 2024-01-21T19:59:01.782 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.789+0000 7fd11a7fc700 1 -- 172.21.15.53:0/1920929903 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd10400dbd0 con 0x7fd11c0724b0 2024-01-21T19:59:01.782 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.789+0000 7fd11a7fc700 1 -- 172.21.15.53:0/1920929903 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd104011c00 con 0x7fd11c0724b0 2024-01-21T19:59:01.782 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.790+0000 7fd121da0700 1 -- 172.21.15.53:0/1920929903 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd11c0724b0 msgr2=0x7fd11c070a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:01.782 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.790+0000 7fd121da0700 1 --2- 172.21.15.53:0/1920929903 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd11c0724b0 0x7fd11c070a50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fd10400b5f0 tx=0x7fd104006ee0).stop 2024-01-21T19:59:01.783 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.790+0000 7fd121da0700 1 -- 172.21.15.53:0/1920929903 shutdown_connections 2024-01-21T19:59:01.784 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.790+0000 7fd121da0700 1 --2- 172.21.15.53:0/1920929903 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd11c0724b0 0x7fd11c070a50 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:01.784 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.790+0000 7fd121da0700 1 -- 172.21.15.53:0/1920929903 >> 172.21.15.53:0/1920929903 conn(0x7fd11c0f74c0 msgr2=0x7fd11c0f98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:01.784 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.790+0000 7fd121da0700 1 -- 172.21.15.53:0/1920929903 shutdown_connections 2024-01-21T19:59:01.784 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.790+0000 7fd121da0700 1 -- 172.21.15.53:0/1920929903 wait complete. 2024-01-21T19:59:01.784 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.791+0000 7fd121da0700 1 Processor -- start 2024-01-21T19:59:01.784 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.791+0000 7fd121da0700 1 -- start start 2024-01-21T19:59:01.784 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.791+0000 7fd121da0700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd11c0724b0 0x7fd11c10cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:01.784 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.791+0000 7fd121da0700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd11c10d1f0 con 0x7fd11c0724b0 2024-01-21T19:59:01.784 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.791+0000 7fd11b7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd11c0724b0 0x7fd11c10cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:01.784 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.792+0000 7fd11b7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd11c0724b0 0x7fd11c10cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:37440/0 (socket says 172.21.15.53:37440) 2024-01-21T19:59:01.784 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.792+0000 7fd11b7fe700 1 -- 172.21.15.53:0/2107007170 learned_addr learned my addr 172.21.15.53:0/2107007170 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:59:01.785 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.792+0000 7fd11b7fe700 1 -- 172.21.15.53:0/2107007170 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd10400b200 con 0x7fd11c0724b0 2024-01-21T19:59:01.785 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.792+0000 7fd11b7fe700 1 --2- 172.21.15.53:0/2107007170 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd11c0724b0 0x7fd11c10cd00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fd10400e040 tx=0x7fd104006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:01.785 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.792+0000 7fd113fff700 1 -- 172.21.15.53:0/2107007170 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd10400d9f0 con 0x7fd11c0724b0 2024-01-21T19:59:01.785 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.792+0000 7fd121da0700 1 -- 172.21.15.53:0/2107007170 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd11c10d3f0 con 0x7fd11c0724b0 2024-01-21T19:59:01.786 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.792+0000 7fd113fff700 1 -- 172.21.15.53:0/2107007170 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd10401a9f0 con 0x7fd11c0724b0 2024-01-21T19:59:01.786 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.792+0000 7fd113fff700 1 -- 172.21.15.53:0/2107007170 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd10401b8c0 con 0x7fd11c0724b0 2024-01-21T19:59:01.786 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.792+0000 7fd121da0700 1 -- 172.21.15.53:0/2107007170 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd11c10d800 con 0x7fd11c0724b0 2024-01-21T19:59:01.786 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.793+0000 7fd113fff700 1 -- 172.21.15.53:0/2107007170 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fd10400f070 con 0x7fd11c0724b0 2024-01-21T19:59:01.786 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.793+0000 7fd113fff700 1 --2- 172.21.15.53:0/2107007170 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7fd0fc02fe60 0x7fd0fc032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:01.786 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.793+0000 7fd113fff700 1 -- 172.21.15.53:0/2107007170 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fd1040415d0 con 0x7fd11c0724b0 2024-01-21T19:59:01.786 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.793+0000 7fd121da0700 1 -- 172.21.15.53:0/2107007170 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd11c101dc0 con 0x7fd11c0724b0 2024-01-21T19:59:01.786 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.793+0000 7fd11affd700 1 --2- 172.21.15.53:0/2107007170 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7fd0fc02fe60 0x7fd0fc032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:01.786 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.794+0000 7fd11affd700 1 --2- 172.21.15.53:0/2107007170 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7fd0fc02fe60 0x7fd0fc032320 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fd10c0096a0 tx=0x7fd10c006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:01.789 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.798+0000 7fd113fff700 1 -- 172.21.15.53:0/2107007170 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd104016070 con 0x7fd11c0724b0 2024-01-21T19:59:01.949 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.957+0000 7fd121da0700 1 -- 172.21.15.53:0/2107007170 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fd11c10e630 con 0x7fd11c0724b0 2024-01-21T19:59:01.952 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.960+0000 7fd113fff700 1 -- 172.21.15.53:0/2107007170 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v8)=0 v8) v1 ==== 142+0+0 (secure 0 0 0) 0x7fd104025460 con 0x7fd11c0724b0 2024-01-21T19:59:01.955 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.962+0000 7fd121da0700 1 -- 172.21.15.53:0/2107007170 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7fd0fc02fe60 msgr2=0x7fd0fc032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:01.955 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.962+0000 7fd121da0700 1 --2- 172.21.15.53:0/2107007170 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7fd0fc02fe60 0x7fd0fc032320 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fd10c0096a0 tx=0x7fd10c006b40).stop 2024-01-21T19:59:01.955 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.962+0000 7fd121da0700 1 -- 172.21.15.53:0/2107007170 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd11c0724b0 msgr2=0x7fd11c10cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:01.955 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.962+0000 7fd121da0700 1 --2- 172.21.15.53:0/2107007170 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd11c0724b0 0x7fd11c10cd00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fd10400e040 tx=0x7fd104006ee0).stop 2024-01-21T19:59:01.955 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.962+0000 7fd121da0700 1 -- 172.21.15.53:0/2107007170 shutdown_connections 2024-01-21T19:59:01.955 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.962+0000 7fd121da0700 1 --2- 172.21.15.53:0/2107007170 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7fd0fc02fe60 0x7fd0fc032320 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:01.955 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.962+0000 7fd121da0700 1 --2- 172.21.15.53:0/2107007170 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd11c0724b0 0x7fd11c10cd00 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:01.955 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.962+0000 7fd121da0700 1 -- 172.21.15.53:0/2107007170 >> 172.21.15.53:0/2107007170 conn(0x7fd11c0f74c0 msgr2=0x7fd11c0f9060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:01.955 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.962+0000 7fd121da0700 1 -- 172.21.15.53:0/2107007170 shutdown_connections 2024-01-21T19:59:01.955 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:01.962+0000 7fd121da0700 1 -- 172.21.15.53:0/2107007170 wait complete. 2024-01-21T19:59:02.347 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/57f6722c-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpw7bt2for:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp987f45g0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2024-01-21T19:59:02.684 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:02 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:59:01.095027+0000 mgr.smithi053.fgsmol (mgr.14144) 7 : audit [DBG] from='client.14156 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T19:59:02.684 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:02 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cephadm 2024-01-21T19:59:01.095542+0000 mgr.smithi053.fgsmol (mgr.14144) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2024-01-21T19:59:02.684 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:02 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:59:01.961583+0000 mon.smithi053 (mon.0) 83 : audit [INF] from='client.? 172.21.15.53:0/2107007170' entity='client.admin' 2024-01-21T19:59:02.697 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.703+0000 7fc74b43a700 1 Processor -- start 2024-01-21T19:59:02.697 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.703+0000 7fc74b43a700 1 -- start start 2024-01-21T19:59:02.697 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.704+0000 7fc74b43a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7440fab80 0x7fc7440fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:02.697 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.704+0000 7fc74b43a700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7440fb490 con 0x7fc7440fab80 2024-01-21T19:59:02.697 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.704+0000 7fc7491d6700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7440fab80 0x7fc7440fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:02.697 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.704+0000 7fc7491d6700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7440fab80 0x7fc7440fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:37454/0 (socket says 172.21.15.53:37454) 2024-01-21T19:59:02.697 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.704+0000 7fc7491d6700 1 -- 172.21.15.53:0/2778999378 learned_addr learned my addr 172.21.15.53:0/2778999378 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:59:02.698 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.704+0000 7fc7491d6700 1 -- 172.21.15.53:0/2778999378 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7440fb5d0 con 0x7fc7440fab80 2024-01-21T19:59:02.698 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.704+0000 7fc7491d6700 1 --2- 172.21.15.53:0/2778999378 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7440fab80 0x7fc7440fafa0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fc73400b5f0 tx=0x7fc734006ee0).ready entity=mon.0 client_cookie=c34bb25730276af6 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:02.698 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.704+0000 7fc733fff700 1 -- 172.21.15.53:0/2778999378 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc73400d600 con 0x7fc7440fab80 2024-01-21T19:59:02.698 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.704+0000 7fc733fff700 1 -- 172.21.15.53:0/2778999378 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc73400dbd0 con 0x7fc7440fab80 2024-01-21T19:59:02.698 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.704+0000 7fc733fff700 1 -- 172.21.15.53:0/2778999378 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc734011c00 con 0x7fc7440fab80 2024-01-21T19:59:02.698 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.705+0000 7fc74b43a700 1 -- 172.21.15.53:0/2778999378 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7440fab80 msgr2=0x7fc7440fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:02.698 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.705+0000 7fc74b43a700 1 --2- 172.21.15.53:0/2778999378 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7440fab80 0x7fc7440fafa0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fc73400b5f0 tx=0x7fc734006ee0).stop 2024-01-21T19:59:02.698 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.705+0000 7fc74b43a700 1 -- 172.21.15.53:0/2778999378 shutdown_connections 2024-01-21T19:59:02.698 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.705+0000 7fc74b43a700 1 --2- 172.21.15.53:0/2778999378 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7440fab80 0x7fc7440fafa0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:02.698 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.705+0000 7fc74b43a700 1 -- 172.21.15.53:0/2778999378 >> 172.21.15.53:0/2778999378 conn(0x7fc7440f63c0 msgr2=0x7fc7440f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:02.698 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.705+0000 7fc74b43a700 1 -- 172.21.15.53:0/2778999378 shutdown_connections 2024-01-21T19:59:02.698 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.705+0000 7fc74b43a700 1 -- 172.21.15.53:0/2778999378 wait complete. 2024-01-21T19:59:02.701 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.706+0000 7fc74b43a700 1 Processor -- start 2024-01-21T19:59:02.701 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.706+0000 7fc74b43a700 1 -- start start 2024-01-21T19:59:02.701 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.706+0000 7fc74b43a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7440fab80 0x7fc74410a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:02.701 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.706+0000 7fc74b43a700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc74410aee0 con 0x7fc7440fab80 2024-01-21T19:59:02.701 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.706+0000 7fc7491d6700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7440fab80 0x7fc74410a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:02.701 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.706+0000 7fc7491d6700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7440fab80 0x7fc74410a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:37458/0 (socket says 172.21.15.53:37458) 2024-01-21T19:59:02.701 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.706+0000 7fc7491d6700 1 -- 172.21.15.53:0/127575785 learned_addr learned my addr 172.21.15.53:0/127575785 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:59:02.701 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.706+0000 7fc7491d6700 1 -- 172.21.15.53:0/127575785 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc73400b200 con 0x7fc7440fab80 2024-01-21T19:59:02.701 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.707+0000 7fc7491d6700 1 --2- 172.21.15.53:0/127575785 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7440fab80 0x7fc74410a9f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fc73400b1d0 tx=0x7fc734006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:02.701 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.707+0000 7fc731ffb700 1 -- 172.21.15.53:0/127575785 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc73401b410 con 0x7fc7440fab80 2024-01-21T19:59:02.701 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.707+0000 7fc731ffb700 1 -- 172.21.15.53:0/127575785 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc73401b9e0 con 0x7fc7440fab80 2024-01-21T19:59:02.701 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.707+0000 7fc74b43a700 1 -- 172.21.15.53:0/127575785 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc74410b0e0 con 0x7fc7440fab80 2024-01-21T19:59:02.701 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.707+0000 7fc731ffb700 1 -- 172.21.15.53:0/127575785 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc73401a840 con 0x7fc7440fab80 2024-01-21T19:59:02.701 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.707+0000 7fc74b43a700 1 -- 172.21.15.53:0/127575785 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc74410b4f0 con 0x7fc7440fab80 2024-01-21T19:59:02.702 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.708+0000 7fc731ffb700 1 -- 172.21.15.53:0/127575785 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fc73400f070 con 0x7fc7440fab80 2024-01-21T19:59:02.702 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.708+0000 7fc731ffb700 1 --2- 172.21.15.53:0/127575785 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7fc72c02feb0 0x7fc72c032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:02.702 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.708+0000 7fc74b43a700 1 -- 172.21.15.53:0/127575785 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc744045ad0 con 0x7fc7440fab80 2024-01-21T19:59:02.702 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.708+0000 7fc731ffb700 1 -- 172.21.15.53:0/127575785 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fc7340413c0 con 0x7fc7440fab80 2024-01-21T19:59:02.702 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.708+0000 7fc7489d5700 1 --2- 172.21.15.53:0/127575785 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7fc72c02feb0 0x7fc72c032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:02.702 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.709+0000 7fc7489d5700 1 --2- 172.21.15.53:0/127575785 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7fc72c02feb0 0x7fc72c032370 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fc7400096a0 tx=0x7fc740006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:02.704 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.712+0000 7fc731ffb700 1 -- 172.21.15.53:0/127575785 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc734019300 con 0x7fc7440fab80 2024-01-21T19:59:02.861 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.869+0000 7fc74b43a700 1 -- 172.21.15.53:0/127575785 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fc7440fc890 con 0x7fc7440fab80 2024-01-21T19:59:02.864 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.872+0000 7fc731ffb700 1 -- 172.21.15.53:0/127575785 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v9)=0 v9) v1 ==== 130+0+0 (secure 0 0 0) 0x7fc734025cb0 con 0x7fc7440fab80 2024-01-21T19:59:02.867 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.875+0000 7fc74b43a700 1 -- 172.21.15.53:0/127575785 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7fc72c02feb0 msgr2=0x7fc72c032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:02.868 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.875+0000 7fc74b43a700 1 --2- 172.21.15.53:0/127575785 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7fc72c02feb0 0x7fc72c032370 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fc7400096a0 tx=0x7fc740006b40).stop 2024-01-21T19:59:02.868 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.875+0000 7fc74b43a700 1 -- 172.21.15.53:0/127575785 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7440fab80 msgr2=0x7fc74410a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:02.868 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.875+0000 7fc74b43a700 1 --2- 172.21.15.53:0/127575785 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7440fab80 0x7fc74410a9f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fc73400b1d0 tx=0x7fc734006ee0).stop 2024-01-21T19:59:02.868 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.875+0000 7fc74b43a700 1 -- 172.21.15.53:0/127575785 shutdown_connections 2024-01-21T19:59:02.868 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.875+0000 7fc74b43a700 1 --2- 172.21.15.53:0/127575785 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7fc72c02feb0 0x7fc72c032370 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:02.868 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.875+0000 7fc74b43a700 1 --2- 172.21.15.53:0/127575785 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7440fab80 0x7fc74410a9f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:02.868 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.875+0000 7fc74b43a700 1 -- 172.21.15.53:0/127575785 >> 172.21.15.53:0/127575785 conn(0x7fc7440f63c0 msgr2=0x7fc744070050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:02.868 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.875+0000 7fc74b43a700 1 -- 172.21.15.53:0/127575785 shutdown_connections 2024-01-21T19:59:02.868 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:02.875+0000 7fc74b43a700 1 -- 172.21.15.53:0/127575785 wait complete. 2024-01-21T19:59:03.272 INFO:teuthology.orchestra.run.smithi053.stderr:Enabling the dashboard module... 2024-01-21T19:59:03.272 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/57f6722c-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpw7bt2for:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp987f45g0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2024-01-21T19:59:03.621 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.628+0000 7f8da9431700 1 Processor -- start 2024-01-21T19:59:03.621 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.629+0000 7f8da9431700 1 -- start start 2024-01-21T19:59:03.621 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.629+0000 7f8da9431700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8da4070430 0x7f8da40fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:03.621 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.629+0000 7f8da9431700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da40fedd0 con 0x7f8da4070430 2024-01-21T19:59:03.621 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.629+0000 7f8da2ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8da4070430 0x7f8da40fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:03.622 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.629+0000 7f8da2ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8da4070430 0x7f8da40fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:37466/0 (socket says 172.21.15.53:37466) 2024-01-21T19:59:03.623 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.629+0000 7f8da2ffd700 1 -- 172.21.15.53:0/3303532725 learned_addr learned my addr 172.21.15.53:0/3303532725 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:59:03.623 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.629+0000 7f8da2ffd700 1 -- 172.21.15.53:0/3303532725 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8da40fef10 con 0x7f8da4070430 2024-01-21T19:59:03.623 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.629+0000 7f8da2ffd700 1 --2- 172.21.15.53:0/3303532725 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8da4070430 0x7f8da40fe8e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f8d9400b5f0 tx=0x7f8d94006ee0).ready entity=mon.0 client_cookie=dfec8716c385551 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:03.623 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.629+0000 7f8da1ffb700 1 -- 172.21.15.53:0/3303532725 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d9400d600 con 0x7f8da4070430 2024-01-21T19:59:03.623 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.630+0000 7f8da1ffb700 1 -- 172.21.15.53:0/3303532725 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f8d9400dbd0 con 0x7f8da4070430 2024-01-21T19:59:03.623 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.630+0000 7f8da1ffb700 1 -- 172.21.15.53:0/3303532725 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d94011c00 con 0x7f8da4070430 2024-01-21T19:59:03.624 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.630+0000 7f8da9431700 1 -- 172.21.15.53:0/3303532725 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8da4070430 msgr2=0x7f8da40fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:03.624 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.630+0000 7f8da9431700 1 --2- 172.21.15.53:0/3303532725 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8da4070430 0x7f8da40fe8e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f8d9400b5f0 tx=0x7f8d94006ee0).stop 2024-01-21T19:59:03.624 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.630+0000 7f8da9431700 1 -- 172.21.15.53:0/3303532725 shutdown_connections 2024-01-21T19:59:03.624 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.630+0000 7f8da9431700 1 --2- 172.21.15.53:0/3303532725 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8da4070430 0x7f8da40fe8e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:03.624 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.630+0000 7f8da9431700 1 -- 172.21.15.53:0/3303532725 >> 172.21.15.53:0/3303532725 conn(0x7f8da40f63c0 msgr2=0x7f8da40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:03.624 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.630+0000 7f8da9431700 1 -- 172.21.15.53:0/3303532725 shutdown_connections 2024-01-21T19:59:03.624 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.630+0000 7f8da9431700 1 -- 172.21.15.53:0/3303532725 wait complete. 2024-01-21T19:59:03.626 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.631+0000 7f8da9431700 1 Processor -- start 2024-01-21T19:59:03.627 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.631+0000 7f8da9431700 1 -- start start 2024-01-21T19:59:03.627 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.632+0000 7f8da9431700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8da4070430 0x7f8da410aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:03.627 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.632+0000 7f8da9431700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da410afb0 con 0x7f8da4070430 2024-01-21T19:59:03.627 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.632+0000 7f8da2ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8da4070430 0x7f8da410aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:03.627 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.632+0000 7f8da2ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8da4070430 0x7f8da410aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:37472/0 (socket says 172.21.15.53:37472) 2024-01-21T19:59:03.627 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.632+0000 7f8da2ffd700 1 -- 172.21.15.53:0/2122137403 learned_addr learned my addr 172.21.15.53:0/2122137403 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:59:03.627 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.632+0000 7f8da2ffd700 1 -- 172.21.15.53:0/2122137403 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d9400b200 con 0x7f8da4070430 2024-01-21T19:59:03.627 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.632+0000 7f8da2ffd700 1 --2- 172.21.15.53:0/2122137403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8da4070430 0x7f8da410aac0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f8d94002f40 tx=0x7f8d94006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:03.627 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.632+0000 7f8d9b7fe700 1 -- 172.21.15.53:0/2122137403 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d94011690 con 0x7f8da4070430 2024-01-21T19:59:03.627 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.632+0000 7f8d9b7fe700 1 -- 172.21.15.53:0/2122137403 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f8d9401cb10 con 0x7f8da4070430 2024-01-21T19:59:03.627 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.632+0000 7f8da9431700 1 -- 172.21.15.53:0/2122137403 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8da410b1b0 con 0x7f8da4070430 2024-01-21T19:59:03.627 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.632+0000 7f8d9b7fe700 1 -- 172.21.15.53:0/2122137403 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d94025a40 con 0x7f8da4070430 2024-01-21T19:59:03.627 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.633+0000 7f8da9431700 1 -- 172.21.15.53:0/2122137403 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8da410b5c0 con 0x7f8da4070430 2024-01-21T19:59:03.627 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.633+0000 7f8d9b7fe700 1 -- 172.21.15.53:0/2122137403 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f8d9402c060 con 0x7f8da4070430 2024-01-21T19:59:03.628 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.633+0000 7f8d9b7fe700 1 --2- 172.21.15.53:0/2122137403 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7f8d8402fe60 0x7f8d84032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:03.628 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.633+0000 7f8d9b7fe700 1 -- 172.21.15.53:0/2122137403 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f8d940450b0 con 0x7f8da4070430 2024-01-21T19:59:03.628 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.634+0000 7f8da27fc700 1 --2- 172.21.15.53:0/2122137403 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7f8d8402fe60 0x7f8d84032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:03.628 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.634+0000 7f8da27fc700 1 --2- 172.21.15.53:0/2122137403 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7f8d8402fe60 0x7f8d84032320 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f8d8c0096a0 tx=0x7f8d8c006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:03.628 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.634+0000 7f8da9431700 1 -- 172.21.15.53:0/2122137403 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8da4059100 con 0x7f8da4070430 2024-01-21T19:59:03.630 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.638+0000 7f8d9b7fe700 1 -- 172.21.15.53:0/2122137403 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8d94016030 con 0x7f8da4070430 2024-01-21T19:59:03.825 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.833+0000 7f8da9431700 1 -- 172.21.15.53:0/2122137403 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f8da40ffb90 con 0x7f8da4070430 2024-01-21T19:59:03.866 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.874+0000 7f8d9b7fe700 1 -- 172.21.15.53:0/2122137403 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f8d94035020 con 0x7f8da4070430 2024-01-21T19:59:03.867 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.875+0000 7f8d9b7fe700 1 -- 172.21.15.53:0/2122137403 <== mon.0 v2:172.21.15.53:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v13) v1 ==== 88+0+0 (secure 0 0 0) 0x7f8d94024630 con 0x7f8da4070430 2024-01-21T19:59:03.870 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.877+0000 7f8da9431700 1 -- 172.21.15.53:0/2122137403 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7f8d8402fe60 msgr2=0x7f8d84032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:03.870 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.877+0000 7f8da9431700 1 --2- 172.21.15.53:0/2122137403 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7f8d8402fe60 0x7f8d84032320 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f8d8c0096a0 tx=0x7f8d8c006b40).stop 2024-01-21T19:59:03.870 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.877+0000 7f8da9431700 1 -- 172.21.15.53:0/2122137403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8da4070430 msgr2=0x7f8da410aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:03.870 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.877+0000 7f8da9431700 1 --2- 172.21.15.53:0/2122137403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8da4070430 0x7f8da410aac0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f8d94002f40 tx=0x7f8d94006ee0).stop 2024-01-21T19:59:03.870 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.877+0000 7f8da9431700 1 -- 172.21.15.53:0/2122137403 shutdown_connections 2024-01-21T19:59:03.870 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.877+0000 7f8da9431700 1 --2- 172.21.15.53:0/2122137403 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7f8d8402fe60 0x7f8d84032320 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:03.870 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.877+0000 7f8da9431700 1 --2- 172.21.15.53:0/2122137403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8da4070430 0x7f8da410aac0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:03.870 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.877+0000 7f8da9431700 1 -- 172.21.15.53:0/2122137403 >> 172.21.15.53:0/2122137403 conn(0x7f8da40f63c0 msgr2=0x7f8da40f7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:03.870 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.877+0000 7f8da9431700 1 -- 172.21.15.53:0/2122137403 shutdown_connections 2024-01-21T19:59:03.870 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:03.877+0000 7f8da9431700 1 -- 172.21.15.53:0/2122137403 wait complete. 2024-01-21T19:59:04.167 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:03 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:59:02.873757+0000 mon.smithi053 (mon.0) 84 : audit [INF] from='client.? 172.21.15.53:0/127575785' entity='client.admin' 2024-01-21T19:59:04.167 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:03 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:59:03.834838+0000 mon.smithi053 (mon.0) 85 : audit [INF] from='client.? 172.21.15.53:0/2122137403' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-01-21T19:59:04.231 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/57f6722c-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpw7bt2for:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp987f45g0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-01-21T19:59:04.641 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.649+0000 7f8ba7569700 1 Processor -- start 2024-01-21T19:59:04.641 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.649+0000 7f8ba7569700 1 -- start start 2024-01-21T19:59:04.641 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.649+0000 7f8ba7569700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ba00703f0 0x7f8ba0100d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:04.644 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.649+0000 7f8ba7569700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba00fc710 con 0x7f8ba00703f0 2024-01-21T19:59:04.644 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.649+0000 7f8ba5305700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ba00703f0 0x7f8ba0100d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:04.644 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.649+0000 7f8ba5305700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ba00703f0 0x7f8ba0100d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:37498/0 (socket says 172.21.15.53:37498) 2024-01-21T19:59:04.644 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.649+0000 7f8ba5305700 1 -- 172.21.15.53:0/32942450 learned_addr learned my addr 172.21.15.53:0/32942450 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:59:04.644 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.650+0000 7f8ba5305700 1 -- 172.21.15.53:0/32942450 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ba0101210 con 0x7f8ba00703f0 2024-01-21T19:59:04.644 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.650+0000 7f8ba5305700 1 --2- 172.21.15.53:0/32942450 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ba00703f0 0x7f8ba0100d20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f8b9c00b5f0 tx=0x7f8b9c006ee0).ready entity=mon.0 client_cookie=32e76bc4a30b6bd0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:04.644 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.650+0000 7f8b8ffff700 1 -- 172.21.15.53:0/32942450 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b9c00d600 con 0x7f8ba00703f0 2024-01-21T19:59:04.644 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.650+0000 7f8b8ffff700 1 -- 172.21.15.53:0/32942450 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f8b9c00dbd0 con 0x7f8ba00703f0 2024-01-21T19:59:04.645 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.650+0000 7f8b8ffff700 1 -- 172.21.15.53:0/32942450 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b9c011c00 con 0x7f8ba00703f0 2024-01-21T19:59:04.645 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.651+0000 7f8ba7569700 1 -- 172.21.15.53:0/32942450 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ba00703f0 msgr2=0x7f8ba0100d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:04.645 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.651+0000 7f8ba7569700 1 --2- 172.21.15.53:0/32942450 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ba00703f0 0x7f8ba0100d20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f8b9c00b5f0 tx=0x7f8b9c006ee0).stop 2024-01-21T19:59:04.645 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.651+0000 7f8ba7569700 1 -- 172.21.15.53:0/32942450 shutdown_connections 2024-01-21T19:59:04.645 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.651+0000 7f8ba7569700 1 --2- 172.21.15.53:0/32942450 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ba00703f0 0x7f8ba0100d20 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:04.645 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.651+0000 7f8ba7569700 1 -- 172.21.15.53:0/32942450 >> 172.21.15.53:0/32942450 conn(0x7f8ba00f63c0 msgr2=0x7f8ba00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:04.645 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.651+0000 7f8ba7569700 1 -- 172.21.15.53:0/32942450 shutdown_connections 2024-01-21T19:59:04.645 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.651+0000 7f8ba7569700 1 -- 172.21.15.53:0/32942450 wait complete. 2024-01-21T19:59:04.645 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.651+0000 7f8ba7569700 1 Processor -- start 2024-01-21T19:59:04.645 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.652+0000 7f8ba7569700 1 -- start start 2024-01-21T19:59:04.645 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.652+0000 7f8ba7569700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ba00703f0 0x7f8ba010ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:04.645 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.652+0000 7f8ba7569700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba010d1e0 con 0x7f8ba00703f0 2024-01-21T19:59:04.645 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.652+0000 7f8ba5305700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ba00703f0 0x7f8ba010ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:04.645 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.652+0000 7f8ba5305700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ba00703f0 0x7f8ba010ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:37512/0 (socket says 172.21.15.53:37512) 2024-01-21T19:59:04.646 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.652+0000 7f8ba5305700 1 -- 172.21.15.53:0/1601369703 learned_addr learned my addr 172.21.15.53:0/1601369703 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:59:04.646 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.652+0000 7f8ba5305700 1 -- 172.21.15.53:0/1601369703 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b9c00b200 con 0x7f8ba00703f0 2024-01-21T19:59:04.646 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.652+0000 7f8ba5305700 1 --2- 172.21.15.53:0/1601369703 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ba00703f0 0x7f8ba010ccf0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f8b9c00e040 tx=0x7f8b9c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:04.647 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.652+0000 7f8b8dffb700 1 -- 172.21.15.53:0/1601369703 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b9c00d9f0 con 0x7f8ba00703f0 2024-01-21T19:59:04.647 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.653+0000 7f8b8dffb700 1 -- 172.21.15.53:0/1601369703 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f8b9c01a9f0 con 0x7f8ba00703f0 2024-01-21T19:59:04.647 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.653+0000 7f8ba7569700 1 -- 172.21.15.53:0/1601369703 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ba010d3e0 con 0x7f8ba00703f0 2024-01-21T19:59:04.647 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.653+0000 7f8b8dffb700 1 -- 172.21.15.53:0/1601369703 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b9c01b8c0 con 0x7f8ba00703f0 2024-01-21T19:59:04.647 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.653+0000 7f8ba7569700 1 -- 172.21.15.53:0/1601369703 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ba010d7f0 con 0x7f8ba00703f0 2024-01-21T19:59:04.647 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.654+0000 7f8ba7569700 1 -- 172.21.15.53:0/1601369703 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ba0045ad0 con 0x7f8ba00703f0 2024-01-21T19:59:04.647 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.654+0000 7f8b8dffb700 1 -- 172.21.15.53:0/1601369703 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f8b9c00f070 con 0x7f8ba00703f0 2024-01-21T19:59:04.647 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.654+0000 7f8b8dffb700 1 --2- 172.21.15.53:0/1601369703 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7f8b8802feb0 0x7f8b88032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:04.647 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.654+0000 7f8b8dffb700 1 -- 172.21.15.53:0/1601369703 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f8b9c041480 con 0x7f8ba00703f0 2024-01-21T19:59:04.647 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.654+0000 7f8ba4b04700 1 -- 172.21.15.53:0/1601369703 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7f8b8802feb0 msgr2=0x7f8b88032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/3045083722 2024-01-21T19:59:04.648 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.654+0000 7f8ba4b04700 1 --2- 172.21.15.53:0/1601369703 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7f8b8802feb0 0x7f8b88032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-21T19:59:04.650 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.658+0000 7f8b8dffb700 1 -- 172.21.15.53:0/1601369703 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8b9c022d50 con 0x7f8ba00703f0 2024-01-21T19:59:04.847 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.855+0000 7f8ba4b04700 1 -- 172.21.15.53:0/1601369703 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7f8b8802feb0 msgr2=0x7f8b88032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/3045083722 2024-01-21T19:59:04.847 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.855+0000 7f8ba4b04700 1 --2- 172.21.15.53:0/1601369703 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7f8b8802feb0 0x7f8b88032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-21T19:59:04.871 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.878+0000 7f8ba7569700 1 -- 172.21.15.53:0/1601369703 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f8ba01065d0 con 0x7f8ba00703f0 2024-01-21T19:59:04.871 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.879+0000 7f8b8dffb700 1 -- 172.21.15.53:0/1601369703 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v13) v1 ==== 56+0+104 (secure 0 0 0) 0x7f8b9c016070 con 0x7f8ba00703f0 2024-01-21T19:59:04.871 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: { 2024-01-21T19:59:04.871 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 13, 2024-01-21T19:59:04.872 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "available": true, 2024-01-21T19:59:04.872 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "active_name": "smithi053.fgsmol", 2024-01-21T19:59:04.872 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_standby": 0 2024-01-21T19:59:04.872 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: } 2024-01-21T19:59:04.876 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.881+0000 7f8ba7569700 1 -- 172.21.15.53:0/1601369703 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7f8b8802feb0 msgr2=0x7f8b88032370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-21T19:59:04.876 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.881+0000 7f8ba7569700 1 --2- 172.21.15.53:0/1601369703 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7f8b8802feb0 0x7f8b88032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T19:59:04.876 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.881+0000 7f8ba7569700 1 -- 172.21.15.53:0/1601369703 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ba00703f0 msgr2=0x7f8ba010ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:04.876 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.881+0000 7f8ba7569700 1 --2- 172.21.15.53:0/1601369703 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ba00703f0 0x7f8ba010ccf0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f8b9c00e040 tx=0x7f8b9c006ee0).stop 2024-01-21T19:59:04.876 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.881+0000 7f8ba7569700 1 -- 172.21.15.53:0/1601369703 shutdown_connections 2024-01-21T19:59:04.876 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.881+0000 7f8ba7569700 1 --2- 172.21.15.53:0/1601369703 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7f8b8802feb0 0x7f8b88032370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T19:59:04.876 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.882+0000 7f8ba7569700 1 --2- 172.21.15.53:0/1601369703 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ba00703f0 0x7f8ba010ccf0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:04.876 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.882+0000 7f8ba7569700 1 -- 172.21.15.53:0/1601369703 >> 172.21.15.53:0/1601369703 conn(0x7f8ba00f63c0 msgr2=0x7f8ba00f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:04.877 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.882+0000 7f8ba7569700 1 -- 172.21.15.53:0/1601369703 shutdown_connections 2024-01-21T19:59:04.877 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:04.882+0000 7f8ba7569700 1 -- 172.21.15.53:0/1601369703 wait complete. 2024-01-21T19:59:05.145 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:04 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:59:03.876743+0000 mon.smithi053 (mon.0) 86 : audit [INF] from='client.? 172.21.15.53:0/2122137403' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-01-21T19:59:05.145 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:04 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:59:03.876828+0000 mon.smithi053 (mon.0) 87 : cluster [DBG] mgrmap e13: smithi053.fgsmol(active, since 7s) 2024-01-21T19:59:05.238 INFO:teuthology.orchestra.run.smithi053.stderr:Waiting for the mgr to restart... 2024-01-21T19:59:05.239 INFO:teuthology.orchestra.run.smithi053.stderr:Waiting for mgr epoch 13... 2024-01-21T19:59:05.239 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/57f6722c-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpw7bt2for:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp987f45g0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-01-21T19:59:05.742 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:05.749+0000 7fb713345700 1 Processor -- start 2024-01-21T19:59:05.742 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:05.749+0000 7fb713345700 1 -- start start 2024-01-21T19:59:05.742 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:05.749+0000 7fb713345700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb70c070430 0x7fb70c0fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:05.743 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:05.749+0000 7fb713345700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb70c0fedd0 con 0x7fb70c070430 2024-01-21T19:59:05.743 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:05.749+0000 7fb7110e1700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb70c070430 0x7fb70c0fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:05.743 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:05.749+0000 7fb7110e1700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb70c070430 0x7fb70c0fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:37518/0 (socket says 172.21.15.53:37518) 2024-01-21T19:59:05.743 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:05.749+0000 7fb7110e1700 1 -- 172.21.15.53:0/2387925715 learned_addr learned my addr 172.21.15.53:0/2387925715 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:59:05.743 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:05.750+0000 7fb7110e1700 1 -- 172.21.15.53:0/2387925715 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb70c0fef10 con 0x7fb70c070430 2024-01-21T19:59:05.743 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:05.750+0000 7fb7110e1700 1 --2- 172.21.15.53:0/2387925715 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb70c070430 0x7fb70c0fe8e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fb70800b5f0 tx=0x7fb708006ee0).ready entity=mon.0 client_cookie=643217841885072d server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:05.744 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:05.750+0000 7fb6fbfff700 1 -- 172.21.15.53:0/2387925715 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb70800d600 con 0x7fb70c070430 2024-01-21T19:59:05.744 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:05.750+0000 7fb6fbfff700 1 -- 172.21.15.53:0/2387925715 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb70800dbd0 con 0x7fb70c070430 2024-01-21T19:59:05.744 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:05.750+0000 7fb6fbfff700 1 -- 172.21.15.53:0/2387925715 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb708011c00 con 0x7fb70c070430 2024-01-21T19:59:05.744 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:05.751+0000 7fb713345700 1 -- 172.21.15.53:0/2387925715 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb70c070430 msgr2=0x7fb70c0fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:05.744 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:05.751+0000 7fb713345700 1 --2- 172.21.15.53:0/2387925715 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb70c070430 0x7fb70c0fe8e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fb70800b5f0 tx=0x7fb708006ee0).stop 2024-01-21T19:59:05.744 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:05.751+0000 7fb713345700 1 -- 172.21.15.53:0/2387925715 shutdown_connections 2024-01-21T19:59:05.744 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:05.751+0000 7fb713345700 1 --2- 172.21.15.53:0/2387925715 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb70c070430 0x7fb70c0fe8e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:05.745 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:05.751+0000 7fb713345700 1 -- 172.21.15.53:0/2387925715 >> 172.21.15.53:0/2387925715 conn(0x7fb70c0f63c0 msgr2=0x7fb70c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:05.745 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:05.751+0000 7fb713345700 1 -- 172.21.15.53:0/2387925715 shutdown_connections 2024-01-21T19:59:05.745 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:05.751+0000 7fb713345700 1 -- 172.21.15.53:0/2387925715 wait complete. 2024-01-21T19:59:05.745 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:05.751+0000 7fb713345700 1 Processor -- start 2024-01-21T19:59:05.745 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:05.751+0000 7fb713345700 1 -- start start 2024-01-21T19:59:05.745 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:05.752+0000 7fb713345700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb70c070430 0x7fb70c10ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:05.745 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:05.752+0000 7fb713345700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb70c10d1c0 con 0x7fb70c070430 2024-01-21T19:59:05.745 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:05.752+0000 7fb7110e1700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb70c070430 0x7fb70c10ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:05.745 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:05.752+0000 7fb7110e1700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb70c070430 0x7fb70c10ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:37534/0 (socket says 172.21.15.53:37534) 2024-01-21T19:59:05.745 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:05.752+0000 7fb7110e1700 1 -- 172.21.15.53:0/2037364005 learned_addr learned my addr 172.21.15.53:0/2037364005 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:59:05.746 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:05.752+0000 7fb7110e1700 1 -- 172.21.15.53:0/2037364005 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb70800b200 con 0x7fb70c070430 2024-01-21T19:59:05.746 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:05.752+0000 7fb7110e1700 1 --2- 172.21.15.53:0/2037364005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb70c070430 0x7fb70c10ccd0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fb70800e040 tx=0x7fb708006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:05.746 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:05.752+0000 7fb6f9ffb700 1 -- 172.21.15.53:0/2037364005 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb70800d9f0 con 0x7fb70c070430 2024-01-21T19:59:05.747 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:05.752+0000 7fb713345700 1 -- 172.21.15.53:0/2037364005 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb70c10d3c0 con 0x7fb70c070430 2024-01-21T19:59:05.747 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:05.752+0000 7fb6f9ffb700 1 -- 172.21.15.53:0/2037364005 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb70801a9f0 con 0x7fb70c070430 2024-01-21T19:59:05.747 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:05.752+0000 7fb6f9ffb700 1 -- 172.21.15.53:0/2037364005 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb70801b8c0 con 0x7fb70c070430 2024-01-21T19:59:05.747 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:05.753+0000 7fb713345700 1 -- 172.21.15.53:0/2037364005 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb70c10d7d0 con 0x7fb70c070430 2024-01-21T19:59:05.747 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:05.754+0000 7fb6f9ffb700 1 -- 172.21.15.53:0/2037364005 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fb70800f070 con 0x7fb70c070430 2024-01-21T19:59:05.747 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:05.754+0000 7fb6f9ffb700 1 --2- 172.21.15.53:0/2037364005 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7fb6f402feb0 0x7fb6f4032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:05.747 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:05.754+0000 7fb6f9ffb700 1 -- 172.21.15.53:0/2037364005 --> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb6f4032a30 con 0x7fb6f402feb0 2024-01-21T19:59:05.747 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:05.754+0000 7fb6f9ffb700 1 -- 172.21.15.53:0/2037364005 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fb708041470 con 0x7fb70c070430 2024-01-21T19:59:05.747 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:05.754+0000 7fb7108e0700 1 -- 172.21.15.53:0/2037364005 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7fb6f402feb0 msgr2=0x7fb6f4032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/3045083722 2024-01-21T19:59:05.747 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:05.754+0000 7fb7108e0700 1 --2- 172.21.15.53:0/2037364005 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7fb6f402feb0 0x7fb6f4032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-21T19:59:05.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:05 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:59:04.880232+0000 mon.smithi053 (mon.0) 88 : audit [DBG] from='client.? 172.21.15.53:0/1601369703' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-01-21T19:59:05.947 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:05.955+0000 7fb7108e0700 1 -- 172.21.15.53:0/2037364005 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7fb6f402feb0 msgr2=0x7fb6f4032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/3045083722 2024-01-21T19:59:05.947 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:05.955+0000 7fb7108e0700 1 --2- 172.21.15.53:0/2037364005 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7fb6f402feb0 0x7fb6f4032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-21T19:59:06.351 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:06.356+0000 7fb7108e0700 1 -- 172.21.15.53:0/2037364005 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7fb6f402feb0 msgr2=0x7fb6f4032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/3045083722 2024-01-21T19:59:06.351 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:06.356+0000 7fb7108e0700 1 --2- 172.21.15.53:0/2037364005 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7fb6f402feb0 0x7fb6f4032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-01-21T19:59:07.149 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:07.157+0000 7fb7108e0700 1 -- 172.21.15.53:0/2037364005 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7fb6f402feb0 msgr2=0x7fb6f4032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/3045083722 2024-01-21T19:59:07.149 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:07.157+0000 7fb7108e0700 1 --2- 172.21.15.53:0/2037364005 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7fb6f402feb0 0x7fb6f4032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-01-21T19:59:08.751 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.758+0000 7fb7108e0700 1 -- 172.21.15.53:0/2037364005 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7fb6f402feb0 msgr2=0x7fb6f4032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/3045083722 2024-01-21T19:59:08.751 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.758+0000 7fb7108e0700 1 --2- 172.21.15.53:0/2037364005 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7fb6f402feb0 0x7fb6f4032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-01-21T19:59:09.113 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.120+0000 7fb6f9ffb700 1 -- 172.21.15.53:0/2037364005 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mgrmap(e 14) v1 ==== 35997+0+0 (secure 0 0 0) 0x7fb70801ba20 con 0x7fb70c070430 2024-01-21T19:59:09.113 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.120+0000 7fb6f9ffb700 1 -- 172.21.15.53:0/2037364005 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7fb6f402feb0 msgr2=0x7fb6f4032370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-21T19:59:09.113 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.120+0000 7fb6f9ffb700 1 --2- 172.21.15.53:0/2037364005 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7fb6f402feb0 0x7fb6f4032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T19:59:10.115 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:10.122+0000 7fb6f9ffb700 1 -- 172.21.15.53:0/2037364005 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7fb708023da0 con 0x7fb70c070430 2024-01-21T19:59:10.115 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:10.122+0000 7fb6f9ffb700 1 --2- 172.21.15.53:0/2037364005 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb6f4033b40 0x7fb6f4035f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:10.115 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:10.122+0000 7fb6f9ffb700 1 -- 172.21.15.53:0/2037364005 --> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb6f4032a30 con 0x7fb6f4033b40 2024-01-21T19:59:10.115 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:10.123+0000 7fb7108e0700 1 --2- 172.21.15.53:0/2037364005 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb6f4033b40 0x7fb6f4035f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:10.116 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:10.123+0000 7fb7108e0700 1 --2- 172.21.15.53:0/2037364005 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb6f4033b40 0x7fb6f4035f30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fb6fc006fd0 tx=0x7fb6fc006cf0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:10.116 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:10.124+0000 7fb6f9ffb700 1 -- 172.21.15.53:0/2037364005 <== mgr.14164 v2:172.21.15.53:6800/3203971228 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7fb6f4032a30 con 0x7fb6f4033b40 2024-01-21T19:59:10.122 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:10.131+0000 7fb713345700 1 -- 172.21.15.53:0/2037364005 --> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fb70c059100 con 0x7fb6f4033b40 2024-01-21T19:59:10.122 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:10.131+0000 7fb6f9ffb700 1 -- 172.21.15.53:0/2037364005 <== mgr.14164 v2:172.21.15.53:6800/3203971228 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fb70c059100 con 0x7fb6f4033b40 2024-01-21T19:59:10.123 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: { 2024-01-21T19:59:10.123 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "mgrmap_epoch": 15, 2024-01-21T19:59:10.124 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "initialized": true 2024-01-21T19:59:10.124 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: } 2024-01-21T19:59:10.124 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:10.131+0000 7fb713345700 1 -- 172.21.15.53:0/2037364005 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb6f4033b40 msgr2=0x7fb6f4035f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:10.124 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:10.131+0000 7fb713345700 1 --2- 172.21.15.53:0/2037364005 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb6f4033b40 0x7fb6f4035f30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fb6fc006fd0 tx=0x7fb6fc006cf0).stop 2024-01-21T19:59:10.125 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:10.131+0000 7fb713345700 1 -- 172.21.15.53:0/2037364005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb70c070430 msgr2=0x7fb70c10ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:10.125 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:10.131+0000 7fb713345700 1 --2- 172.21.15.53:0/2037364005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb70c070430 0x7fb70c10ccd0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fb70800e040 tx=0x7fb708006ee0).stop 2024-01-21T19:59:10.125 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:10.131+0000 7fb713345700 1 -- 172.21.15.53:0/2037364005 shutdown_connections 2024-01-21T19:59:10.125 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:10.131+0000 7fb713345700 1 --2- 172.21.15.53:0/2037364005 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb6f4033b40 0x7fb6f4035f30 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:10.125 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:10.131+0000 7fb713345700 1 --2- 172.21.15.53:0/2037364005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb70c070430 0x7fb70c10ccd0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:10.125 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:10.131+0000 7fb713345700 1 --2- 172.21.15.53:0/2037364005 >> [v2:172.21.15.53:6800/3045083722,v1:172.21.15.53:6801/3045083722] conn(0x7fb6f402feb0 0x7fb6f4032370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T19:59:10.125 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:10.132+0000 7fb713345700 1 -- 172.21.15.53:0/2037364005 >> 172.21.15.53:0/2037364005 conn(0x7fb70c0f63c0 msgr2=0x7fb70c0f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:10.125 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:10.132+0000 7fb713345700 1 -- 172.21.15.53:0/2037364005 shutdown_connections 2024-01-21T19:59:10.125 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:10.132+0000 7fb713345700 1 -- 172.21.15.53:0/2037364005 wait complete. 2024-01-21T19:59:10.434 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:10 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:59:09.070024+0000 mon.smithi053 (mon.0) 89 : cluster [INF] Active manager daemon smithi053.fgsmol restarted 2024-01-21T19:59:10.434 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:10 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:59:09.070395+0000 mon.smithi053 (mon.0) 90 : cluster [INF] Activating manager daemon smithi053.fgsmol 2024-01-21T19:59:10.434 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:10 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:59:09.071922+0000 mon.smithi053 (mon.0) 91 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-01-21T19:59:10.435 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:10 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:59:09.122748+0000 mon.smithi053 (mon.0) 92 : cluster [DBG] mgrmap e14: smithi053.fgsmol(active, starting, since 0.0524338s) 2024-01-21T19:59:10.435 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:10 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:59:09.123553+0000 mon.smithi053 (mon.0) 93 : audit [DBG] from='mgr.14164 172.21.15.53:0/2331671680' entity='mgr.smithi053.fgsmol' cmd=[{"prefix": "mon metadata", "id": "smithi053"}]: dispatch 2024-01-21T19:59:10.435 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:10 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:59:09.124190+0000 mon.smithi053 (mon.0) 94 : audit [DBG] from='mgr.14164 172.21.15.53:0/2331671680' entity='mgr.smithi053.fgsmol' cmd=[{"prefix": "mgr metadata", "who": "smithi053.fgsmol", "id": "smithi053.fgsmol"}]: dispatch 2024-01-21T19:59:10.435 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:10 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:59:09.125038+0000 mon.smithi053 (mon.0) 95 : audit [DBG] from='mgr.14164 172.21.15.53:0/2331671680' entity='mgr.smithi053.fgsmol' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-21T19:59:10.435 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:10 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:59:09.125184+0000 mon.smithi053 (mon.0) 96 : audit [DBG] from='mgr.14164 172.21.15.53:0/2331671680' entity='mgr.smithi053.fgsmol' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-21T19:59:10.435 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:10 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:59:09.125331+0000 mon.smithi053 (mon.0) 97 : audit [DBG] from='mgr.14164 172.21.15.53:0/2331671680' entity='mgr.smithi053.fgsmol' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-21T19:59:10.435 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:10 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:59:09.138615+0000 mon.smithi053 (mon.0) 98 : cluster [INF] Manager daemon smithi053.fgsmol is now available 2024-01-21T19:59:10.435 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:10 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:59:09.242175+0000 mon.smithi053 (mon.0) 99 : audit [DBG] from='mgr.14164 172.21.15.53:0/2331671680' entity='mgr.smithi053.fgsmol' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T19:59:10.435 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:10 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:59:09.245068+0000 mon.smithi053 (mon.0) 100 : audit [DBG] from='mgr.14164 172.21.15.53:0/2331671680' entity='mgr.smithi053.fgsmol' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T19:59:10.435 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:10 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:59:09.261028+0000 mon.smithi053 (mon.0) 101 : audit [INF] from='mgr.14164 172.21.15.53:0/2331671680' entity='mgr.smithi053.fgsmol' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fgsmol/mirror_snapshot_schedule"}]: dispatch 2024-01-21T19:59:10.436 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:10 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:59:09.269169+0000 mon.smithi053 (mon.0) 102 : audit [INF] from='mgr.14164 172.21.15.53:0/2331671680' entity='mgr.smithi053.fgsmol' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fgsmol/trash_purge_schedule"}]: dispatch 2024-01-21T19:59:10.631 INFO:teuthology.orchestra.run.smithi053.stderr:mgr epoch 13 is available 2024-01-21T19:59:10.631 INFO:teuthology.orchestra.run.smithi053.stderr:Generating a dashboard self-signed certificate... 2024-01-21T19:59:10.631 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/57f6722c-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpw7bt2for:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp987f45g0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2024-01-21T19:59:11.012 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.018+0000 7fd39b0b3700 1 Processor -- start 2024-01-21T19:59:11.012 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.018+0000 7fd39b0b3700 1 -- start start 2024-01-21T19:59:11.013 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.018+0000 7fd39b0b3700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd394070390 0x7fd394100d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:11.013 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.018+0000 7fd39b0b3700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3940fc770 con 0x7fd394070390 2024-01-21T19:59:11.013 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.019+0000 7fd398e4f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd394070390 0x7fd394100d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:11.013 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.019+0000 7fd398e4f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd394070390 0x7fd394100d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:40040/0 (socket says 172.21.15.53:40040) 2024-01-21T19:59:11.013 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.019+0000 7fd398e4f700 1 -- 172.21.15.53:0/927971034 learned_addr learned my addr 172.21.15.53:0/927971034 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:59:11.013 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.019+0000 7fd398e4f700 1 -- 172.21.15.53:0/927971034 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd394101270 con 0x7fd394070390 2024-01-21T19:59:11.013 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.019+0000 7fd398e4f700 1 --2- 172.21.15.53:0/927971034 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd394070390 0x7fd394100d80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fd38800b5f0 tx=0x7fd388006ee0).ready entity=mon.0 client_cookie=81abbdd1ddc9639c server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:11.013 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.019+0000 7fd3937fe700 1 -- 172.21.15.53:0/927971034 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd38800d600 con 0x7fd394070390 2024-01-21T19:59:11.013 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.020+0000 7fd3937fe700 1 -- 172.21.15.53:0/927971034 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd38800dbd0 con 0x7fd394070390 2024-01-21T19:59:11.013 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.020+0000 7fd3937fe700 1 -- 172.21.15.53:0/927971034 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd388011c00 con 0x7fd394070390 2024-01-21T19:59:11.013 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.020+0000 7fd39b0b3700 1 -- 172.21.15.53:0/927971034 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd394070390 msgr2=0x7fd394100d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:11.013 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.020+0000 7fd39b0b3700 1 --2- 172.21.15.53:0/927971034 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd394070390 0x7fd394100d80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fd38800b5f0 tx=0x7fd388006ee0).stop 2024-01-21T19:59:11.013 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.020+0000 7fd39b0b3700 1 -- 172.21.15.53:0/927971034 shutdown_connections 2024-01-21T19:59:11.013 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.020+0000 7fd39b0b3700 1 --2- 172.21.15.53:0/927971034 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd394070390 0x7fd394100d80 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:11.014 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.020+0000 7fd39b0b3700 1 -- 172.21.15.53:0/927971034 >> 172.21.15.53:0/927971034 conn(0x7fd3940f63c0 msgr2=0x7fd3940f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:11.014 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.020+0000 7fd39b0b3700 1 -- 172.21.15.53:0/927971034 shutdown_connections 2024-01-21T19:59:11.014 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.020+0000 7fd39b0b3700 1 -- 172.21.15.53:0/927971034 wait complete. 2024-01-21T19:59:11.014 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.021+0000 7fd39b0b3700 1 Processor -- start 2024-01-21T19:59:11.014 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.021+0000 7fd39b0b3700 1 -- start start 2024-01-21T19:59:11.016 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.021+0000 7fd39b0b3700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd394070390 0x7fd39410cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:11.016 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.021+0000 7fd39b0b3700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd39410d110 con 0x7fd394070390 2024-01-21T19:59:11.016 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.021+0000 7fd398e4f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd394070390 0x7fd39410cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:11.016 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.021+0000 7fd398e4f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd394070390 0x7fd39410cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:40052/0 (socket says 172.21.15.53:40052) 2024-01-21T19:59:11.016 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.021+0000 7fd398e4f700 1 -- 172.21.15.53:0/2910273114 learned_addr learned my addr 172.21.15.53:0/2910273114 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:59:11.016 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.021+0000 7fd398e4f700 1 -- 172.21.15.53:0/2910273114 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd38800b200 con 0x7fd394070390 2024-01-21T19:59:11.016 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.021+0000 7fd398e4f700 1 --2- 172.21.15.53:0/2910273114 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd394070390 0x7fd39410cc20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fd38800b190 tx=0x7fd388006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:11.016 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.022+0000 7fd3917fa700 1 -- 172.21.15.53:0/2910273114 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd38801b410 con 0x7fd394070390 2024-01-21T19:59:11.017 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.022+0000 7fd3917fa700 1 -- 172.21.15.53:0/2910273114 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd38801b9e0 con 0x7fd394070390 2024-01-21T19:59:11.017 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.022+0000 7fd39b0b3700 1 -- 172.21.15.53:0/2910273114 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd39410d310 con 0x7fd394070390 2024-01-21T19:59:11.017 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.022+0000 7fd3917fa700 1 -- 172.21.15.53:0/2910273114 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd38801a880 con 0x7fd394070390 2024-01-21T19:59:11.017 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.022+0000 7fd39b0b3700 1 -- 172.21.15.53:0/2910273114 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd39410d720 con 0x7fd394070390 2024-01-21T19:59:11.017 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.022+0000 7fd3917fa700 1 -- 172.21.15.53:0/2910273114 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7fd38800f070 con 0x7fd394070390 2024-01-21T19:59:11.017 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.023+0000 7fd3917fa700 1 --2- 172.21.15.53:0/2910273114 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fd37802fda0 0x7fd378032260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:11.017 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.023+0000 7fd3917fa700 1 -- 172.21.15.53:0/2910273114 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd3880412f0 con 0x7fd394070390 2024-01-21T19:59:11.017 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.023+0000 7fd393fff700 1 --2- 172.21.15.53:0/2910273114 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fd37802fda0 0x7fd378032260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:11.017 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.023+0000 7fd393fff700 1 --2- 172.21.15.53:0/2910273114 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fd37802fda0 0x7fd378032260 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fd3800096a0 tx=0x7fd380006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:11.017 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.023+0000 7fd39b0b3700 1 -- 172.21.15.53:0/2910273114 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd394045ad0 con 0x7fd394070390 2024-01-21T19:59:11.019 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.027+0000 7fd3917fa700 1 -- 172.21.15.53:0/2910273114 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd388025460 con 0x7fd394070390 2024-01-21T19:59:11.117 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.125+0000 7fd3917fa700 1 -- 172.21.15.53:0/2910273114 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fd3880257e0 con 0x7fd394070390 2024-01-21T19:59:11.185 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.193+0000 7fd39b0b3700 1 -- 172.21.15.53:0/2910273114 --> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fd394106b00 con 0x7fd37802fda0 2024-01-21T19:59:11.225 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.233+0000 7fd3917fa700 1 -- 172.21.15.53:0/2910273114 <== mgr.14164 v2:172.21.15.53:6800/3203971228 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fd394106b00 con 0x7fd37802fda0 2024-01-21T19:59:11.225 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: Self-signed certificate created 2024-01-21T19:59:11.228 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.235+0000 7fd39b0b3700 1 -- 172.21.15.53:0/2910273114 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fd37802fda0 msgr2=0x7fd378032260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:11.228 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.235+0000 7fd39b0b3700 1 --2- 172.21.15.53:0/2910273114 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fd37802fda0 0x7fd378032260 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fd3800096a0 tx=0x7fd380006b40).stop 2024-01-21T19:59:11.228 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.235+0000 7fd39b0b3700 1 -- 172.21.15.53:0/2910273114 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd394070390 msgr2=0x7fd39410cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:11.228 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.235+0000 7fd39b0b3700 1 --2- 172.21.15.53:0/2910273114 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd394070390 0x7fd39410cc20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fd38800b190 tx=0x7fd388006ee0).stop 2024-01-21T19:59:11.229 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.236+0000 7fd39b0b3700 1 -- 172.21.15.53:0/2910273114 shutdown_connections 2024-01-21T19:59:11.229 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.236+0000 7fd39b0b3700 1 --2- 172.21.15.53:0/2910273114 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fd37802fda0 0x7fd378032260 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:11.229 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.236+0000 7fd39b0b3700 1 --2- 172.21.15.53:0/2910273114 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd394070390 0x7fd39410cc20 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:11.229 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.236+0000 7fd39b0b3700 1 -- 172.21.15.53:0/2910273114 >> 172.21.15.53:0/2910273114 conn(0x7fd3940f63c0 msgr2=0x7fd3940f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:11.229 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.236+0000 7fd39b0b3700 1 -- 172.21.15.53:0/2910273114 shutdown_connections 2024-01-21T19:59:11.229 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.236+0000 7fd39b0b3700 1 -- 172.21.15.53:0/2910273114 wait complete. 2024-01-21T19:59:11.378 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:11 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:59:10.125254+0000 mgr.smithi053.fgsmol (mgr.14164) 1 : audit [DBG] from='client.14168 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-01-21T19:59:11.378 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:11 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:59:10.126153+0000 mon.smithi053 (mon.0) 103 : cluster [DBG] mgrmap e15: smithi053.fgsmol(active, since 1.05583s) 2024-01-21T19:59:11.378 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:11 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:59:10.132412+0000 mgr.smithi053.fgsmol (mgr.14164) 2 : audit [DBG] from='client.14168 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-01-21T19:59:11.588 INFO:teuthology.orchestra.run.smithi053.stderr:Creating initial admin user... 2024-01-21T19:59:11.589 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/57f6722c-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpw7bt2for:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp987f45g0:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpauncjfok:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2024-01-21T19:59:11.914 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.921+0000 7fddb28e3700 1 Processor -- start 2024-01-21T19:59:11.914 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.921+0000 7fddb28e3700 1 -- start start 2024-01-21T19:59:11.914 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.921+0000 7fddb28e3700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddac0703f0 0x7fddac0fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:11.914 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.921+0000 7fddb28e3700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddac0fee10 con 0x7fddac0703f0 2024-01-21T19:59:11.915 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.921+0000 7fddabfff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddac0703f0 0x7fddac0fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:11.915 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.921+0000 7fddabfff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddac0703f0 0x7fddac0fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:40066/0 (socket says 172.21.15.53:40066) 2024-01-21T19:59:11.915 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.921+0000 7fddabfff700 1 -- 172.21.15.53:0/1797831533 learned_addr learned my addr 172.21.15.53:0/1797831533 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:59:11.915 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.921+0000 7fddabfff700 1 -- 172.21.15.53:0/1797831533 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddac0fef50 con 0x7fddac0703f0 2024-01-21T19:59:11.915 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.921+0000 7fddabfff700 1 --2- 172.21.15.53:0/1797831533 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddac0703f0 0x7fddac0fe920 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fdd9c00b5f0 tx=0x7fdd9c006ee0).ready entity=mon.0 client_cookie=ed32bcfd87641f38 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:11.915 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.921+0000 7fddaaffd700 1 -- 172.21.15.53:0/1797831533 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd9c00d600 con 0x7fddac0703f0 2024-01-21T19:59:11.915 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.921+0000 7fddaaffd700 1 -- 172.21.15.53:0/1797831533 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fdd9c00dbd0 con 0x7fddac0703f0 2024-01-21T19:59:11.915 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.922+0000 7fddaaffd700 1 -- 172.21.15.53:0/1797831533 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd9c011c00 con 0x7fddac0703f0 2024-01-21T19:59:11.915 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.922+0000 7fddb28e3700 1 -- 172.21.15.53:0/1797831533 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddac0703f0 msgr2=0x7fddac0fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:11.915 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.922+0000 7fddb28e3700 1 --2- 172.21.15.53:0/1797831533 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddac0703f0 0x7fddac0fe920 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fdd9c00b5f0 tx=0x7fdd9c006ee0).stop 2024-01-21T19:59:11.915 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.922+0000 7fddb28e3700 1 -- 172.21.15.53:0/1797831533 shutdown_connections 2024-01-21T19:59:11.918 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.922+0000 7fddb28e3700 1 --2- 172.21.15.53:0/1797831533 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddac0703f0 0x7fddac0fe920 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:11.918 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.922+0000 7fddb28e3700 1 -- 172.21.15.53:0/1797831533 >> 172.21.15.53:0/1797831533 conn(0x7fddac0f63c0 msgr2=0x7fddac0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:11.918 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.922+0000 7fddb28e3700 1 -- 172.21.15.53:0/1797831533 shutdown_connections 2024-01-21T19:59:11.918 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.922+0000 7fddb28e3700 1 -- 172.21.15.53:0/1797831533 wait complete. 2024-01-21T19:59:11.918 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.923+0000 7fddb28e3700 1 Processor -- start 2024-01-21T19:59:11.918 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.923+0000 7fddb28e3700 1 -- start start 2024-01-21T19:59:11.918 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.923+0000 7fddb28e3700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddac0703f0 0x7fddac10aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:11.918 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.923+0000 7fddb28e3700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddac10af50 con 0x7fddac0703f0 2024-01-21T19:59:11.919 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.923+0000 7fddabfff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddac0703f0 0x7fddac10aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:11.919 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.923+0000 7fddabfff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddac0703f0 0x7fddac10aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:40082/0 (socket says 172.21.15.53:40082) 2024-01-21T19:59:11.919 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.923+0000 7fddabfff700 1 -- 172.21.15.53:0/2504784137 learned_addr learned my addr 172.21.15.53:0/2504784137 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:59:11.919 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.924+0000 7fddabfff700 1 -- 172.21.15.53:0/2504784137 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd9c00b200 con 0x7fddac0703f0 2024-01-21T19:59:11.919 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.924+0000 7fddabfff700 1 --2- 172.21.15.53:0/2504784137 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddac0703f0 0x7fddac10aa60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fdd9c00e040 tx=0x7fdd9c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:11.919 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.924+0000 7fdda8ff9700 1 -- 172.21.15.53:0/2504784137 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd9c00d9f0 con 0x7fddac0703f0 2024-01-21T19:59:11.919 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.924+0000 7fdda8ff9700 1 -- 172.21.15.53:0/2504784137 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fdd9c01a9f0 con 0x7fddac0703f0 2024-01-21T19:59:11.919 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.924+0000 7fddb28e3700 1 -- 172.21.15.53:0/2504784137 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddac10b150 con 0x7fddac0703f0 2024-01-21T19:59:11.919 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.924+0000 7fdda8ff9700 1 -- 172.21.15.53:0/2504784137 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd9c01b8c0 con 0x7fddac0703f0 2024-01-21T19:59:11.919 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.924+0000 7fddb28e3700 1 -- 172.21.15.53:0/2504784137 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddac10b560 con 0x7fddac0703f0 2024-01-21T19:59:11.919 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.925+0000 7fddb28e3700 1 -- 172.21.15.53:0/2504784137 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddac045ad0 con 0x7fddac0703f0 2024-01-21T19:59:11.919 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.925+0000 7fdda8ff9700 1 -- 172.21.15.53:0/2504784137 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fdd9c00f070 con 0x7fddac0703f0 2024-01-21T19:59:11.919 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.925+0000 7fdda8ff9700 1 --2- 172.21.15.53:0/2504784137 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fdd9802feb0 0x7fdd98032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:11.919 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.926+0000 7fdda8ff9700 1 -- 172.21.15.53:0/2504784137 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdd9c041580 con 0x7fddac0703f0 2024-01-21T19:59:11.919 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.926+0000 7fddab7fe700 1 --2- 172.21.15.53:0/2504784137 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fdd9802feb0 0x7fdd98032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:11.920 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.926+0000 7fddab7fe700 1 --2- 172.21.15.53:0/2504784137 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fdd9802feb0 0x7fdd98032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fdda00096a0 tx=0x7fdda0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:11.921 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.930+0000 7fdda8ff9700 1 -- 172.21.15.53:0/2504784137 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdd9c0224f0 con 0x7fddac0703f0 2024-01-21T19:59:12.095 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.103+0000 7fddb28e3700 1 -- 172.21.15.53:0/2504784137 --> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fddac058f20 con 0x7fdd9802feb0 2024-01-21T19:59:12.340 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.348+0000 7fdda8ff9700 1 -- 172.21.15.53:0/2504784137 <== mgr.14164 v2:172.21.15.53:6800/3203971228 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7fddac058f20 con 0x7fdd9802feb0 2024-01-21T19:59:12.340 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$vI29B7e0tLQEKhZqcJ9NjOGsJnf3uStGnqdoNwX1YHT3pZ8Dr7Y1m", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1705867152, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-01-21T19:59:12.342 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.349+0000 7fddb28e3700 1 -- 172.21.15.53:0/2504784137 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fdd9802feb0 msgr2=0x7fdd98032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:12.342 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.349+0000 7fddb28e3700 1 --2- 172.21.15.53:0/2504784137 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fdd9802feb0 0x7fdd98032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fdda00096a0 tx=0x7fdda0006b40).stop 2024-01-21T19:59:12.342 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.349+0000 7fddb28e3700 1 -- 172.21.15.53:0/2504784137 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddac0703f0 msgr2=0x7fddac10aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:12.342 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.349+0000 7fddb28e3700 1 --2- 172.21.15.53:0/2504784137 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddac0703f0 0x7fddac10aa60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fdd9c00e040 tx=0x7fdd9c006ee0).stop 2024-01-21T19:59:12.342 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.350+0000 7fddb28e3700 1 -- 172.21.15.53:0/2504784137 shutdown_connections 2024-01-21T19:59:12.342 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.350+0000 7fddb28e3700 1 --2- 172.21.15.53:0/2504784137 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fdd9802feb0 0x7fdd98032370 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:12.342 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.350+0000 7fddb28e3700 1 --2- 172.21.15.53:0/2504784137 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddac0703f0 0x7fddac10aa60 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:12.343 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.350+0000 7fddb28e3700 1 -- 172.21.15.53:0/2504784137 >> 172.21.15.53:0/2504784137 conn(0x7fddac0f63c0 msgr2=0x7fddac0f7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:12.343 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.350+0000 7fddb28e3700 1 -- 172.21.15.53:0/2504784137 shutdown_connections 2024-01-21T19:59:12.343 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.350+0000 7fddb28e3700 1 -- 172.21.15.53:0/2504784137 wait complete. 2024-01-21T19:59:12.434 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:12 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:59:11.127635+0000 mon.smithi053 (mon.0) 104 : cluster [DBG] mgrmap e16: smithi053.fgsmol(active, since 2s) 2024-01-21T19:59:12.434 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:12 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:59:11.195303+0000 mgr.smithi053.fgsmol (mgr.14164) 3 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T19:59:12.434 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:12 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:59:11.233049+0000 mon.smithi053 (mon.0) 105 : audit [INF] from='mgr.14164 172.21.15.53:0/2331671680' entity='mgr.smithi053.fgsmol' 2024-01-21T19:59:12.434 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:12 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:59:11.234840+0000 mon.smithi053 (mon.0) 106 : audit [INF] from='mgr.14164 172.21.15.53:0/2331671680' entity='mgr.smithi053.fgsmol' 2024-01-21T19:59:12.688 INFO:teuthology.orchestra.run.smithi053.stderr:Fetching dashboard port number... 2024-01-21T19:59:12.688 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/57f6722c-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpw7bt2for:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp987f45g0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2024-01-21T19:59:13.058 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.062+0000 7fa046cfc700 1 Processor -- start 2024-01-21T19:59:13.058 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.063+0000 7fa046cfc700 1 -- start start 2024-01-21T19:59:13.058 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.063+0000 7fa046cfc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa040070430 0x7fa0400fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:13.058 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.063+0000 7fa046cfc700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0400fedd0 con 0x7fa040070430 2024-01-21T19:59:13.058 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.063+0000 7fa044a98700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa040070430 0x7fa0400fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:13.058 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.063+0000 7fa044a98700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa040070430 0x7fa0400fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:40092/0 (socket says 172.21.15.53:40092) 2024-01-21T19:59:13.059 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.063+0000 7fa044a98700 1 -- 172.21.15.53:0/1981139247 learned_addr learned my addr 172.21.15.53:0/1981139247 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:59:13.059 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.063+0000 7fa044a98700 1 -- 172.21.15.53:0/1981139247 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0400fef10 con 0x7fa040070430 2024-01-21T19:59:13.059 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.063+0000 7fa044a98700 1 --2- 172.21.15.53:0/1981139247 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa040070430 0x7fa0400fe8e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fa03400b5f0 tx=0x7fa034006ee0).ready entity=mon.0 client_cookie=1f19a0381a659b63 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:13.059 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.063+0000 7fa03f7fe700 1 -- 172.21.15.53:0/1981139247 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa03400d600 con 0x7fa040070430 2024-01-21T19:59:13.059 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.064+0000 7fa03f7fe700 1 -- 172.21.15.53:0/1981139247 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa03400dbd0 con 0x7fa040070430 2024-01-21T19:59:13.059 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.064+0000 7fa03f7fe700 1 -- 172.21.15.53:0/1981139247 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa034011c00 con 0x7fa040070430 2024-01-21T19:59:13.059 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.064+0000 7fa046cfc700 1 -- 172.21.15.53:0/1981139247 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa040070430 msgr2=0x7fa0400fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:13.059 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.064+0000 7fa046cfc700 1 --2- 172.21.15.53:0/1981139247 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa040070430 0x7fa0400fe8e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fa03400b5f0 tx=0x7fa034006ee0).stop 2024-01-21T19:59:13.059 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.064+0000 7fa046cfc700 1 -- 172.21.15.53:0/1981139247 shutdown_connections 2024-01-21T19:59:13.059 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.064+0000 7fa046cfc700 1 --2- 172.21.15.53:0/1981139247 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa040070430 0x7fa0400fe8e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:13.059 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.064+0000 7fa046cfc700 1 -- 172.21.15.53:0/1981139247 >> 172.21.15.53:0/1981139247 conn(0x7fa0400f63c0 msgr2=0x7fa0400f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:13.059 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.064+0000 7fa046cfc700 1 -- 172.21.15.53:0/1981139247 shutdown_connections 2024-01-21T19:59:13.060 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.064+0000 7fa046cfc700 1 -- 172.21.15.53:0/1981139247 wait complete. 2024-01-21T19:59:13.060 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.065+0000 7fa046cfc700 1 Processor -- start 2024-01-21T19:59:13.060 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.065+0000 7fa046cfc700 1 -- start start 2024-01-21T19:59:13.060 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.065+0000 7fa046cfc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa040070430 0x7fa04010aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:13.060 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.065+0000 7fa046cfc700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa04010aef0 con 0x7fa040070430 2024-01-21T19:59:13.060 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.065+0000 7fa044a98700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa040070430 0x7fa04010aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:13.060 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.065+0000 7fa044a98700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa040070430 0x7fa04010aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:40098/0 (socket says 172.21.15.53:40098) 2024-01-21T19:59:13.060 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.065+0000 7fa044a98700 1 -- 172.21.15.53:0/996831859 learned_addr learned my addr 172.21.15.53:0/996831859 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:59:13.061 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.066+0000 7fa044a98700 1 -- 172.21.15.53:0/996831859 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa03400b200 con 0x7fa040070430 2024-01-21T19:59:13.062 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.066+0000 7fa044a98700 1 --2- 172.21.15.53:0/996831859 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa040070430 0x7fa04010aa00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fa03400b190 tx=0x7fa034006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:13.062 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.066+0000 7fa03d7fa700 1 -- 172.21.15.53:0/996831859 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa03401b410 con 0x7fa040070430 2024-01-21T19:59:13.062 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.066+0000 7fa03d7fa700 1 -- 172.21.15.53:0/996831859 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa03401b9e0 con 0x7fa040070430 2024-01-21T19:59:13.062 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.066+0000 7fa03d7fa700 1 -- 172.21.15.53:0/996831859 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa03401a880 con 0x7fa040070430 2024-01-21T19:59:13.062 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.066+0000 7fa046cfc700 1 -- 172.21.15.53:0/996831859 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa04010b0f0 con 0x7fa040070430 2024-01-21T19:59:13.062 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.066+0000 7fa046cfc700 1 -- 172.21.15.53:0/996831859 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa04010b500 con 0x7fa040070430 2024-01-21T19:59:13.062 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.067+0000 7fa03d7fa700 1 -- 172.21.15.53:0/996831859 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fa03400f070 con 0x7fa040070430 2024-01-21T19:59:13.062 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.067+0000 7fa03d7fa700 1 --2- 172.21.15.53:0/996831859 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa02402ff00 0x7fa0240323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:13.062 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.067+0000 7fa03d7fa700 1 -- 172.21.15.53:0/996831859 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa0340414c0 con 0x7fa040070430 2024-01-21T19:59:13.062 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.067+0000 7fa03ffff700 1 --2- 172.21.15.53:0/996831859 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa02402ff00 0x7fa0240323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:13.062 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.067+0000 7fa046cfc700 1 -- 172.21.15.53:0/996831859 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa040059100 con 0x7fa040070430 2024-01-21T19:59:13.062 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.067+0000 7fa03ffff700 1 --2- 172.21.15.53:0/996831859 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa02402ff00 0x7fa0240323c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fa02c0096a0 tx=0x7fa02c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:13.063 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.071+0000 7fa03d7fa700 1 -- 172.21.15.53:0/996831859 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa034016070 con 0x7fa040070430 2024-01-21T19:59:13.222 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.230+0000 7fa046cfc700 1 -- 172.21.15.53:0/996831859 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fa0400ffb90 con 0x7fa040070430 2024-01-21T19:59:13.222 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.230+0000 7fa03d7fa700 1 -- 172.21.15.53:0/996831859 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v9) v1 ==== 112+0+5 (secure 0 0 0) 0x7fa034025460 con 0x7fa040070430 2024-01-21T19:59:13.222 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 8443 2024-01-21T19:59:13.224 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.232+0000 7fa046cfc700 1 -- 172.21.15.53:0/996831859 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa02402ff00 msgr2=0x7fa0240323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:13.224 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.232+0000 7fa046cfc700 1 --2- 172.21.15.53:0/996831859 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa02402ff00 0x7fa0240323c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fa02c0096a0 tx=0x7fa02c006b40).stop 2024-01-21T19:59:13.224 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.232+0000 7fa046cfc700 1 -- 172.21.15.53:0/996831859 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa040070430 msgr2=0x7fa04010aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:13.225 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.232+0000 7fa046cfc700 1 --2- 172.21.15.53:0/996831859 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa040070430 0x7fa04010aa00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fa03400b190 tx=0x7fa034006ee0).stop 2024-01-21T19:59:13.225 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.232+0000 7fa046cfc700 1 -- 172.21.15.53:0/996831859 shutdown_connections 2024-01-21T19:59:13.225 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.232+0000 7fa046cfc700 1 --2- 172.21.15.53:0/996831859 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa02402ff00 0x7fa0240323c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:13.225 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.232+0000 7fa046cfc700 1 --2- 172.21.15.53:0/996831859 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa040070430 0x7fa04010aa00 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:13.225 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.232+0000 7fa046cfc700 1 -- 172.21.15.53:0/996831859 >> 172.21.15.53:0/996831859 conn(0x7fa0400f63c0 msgr2=0x7fa0400f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:13.225 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.232+0000 7fa046cfc700 1 -- 172.21.15.53:0/996831859 shutdown_connections 2024-01-21T19:59:13.225 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-01-21T19:59:13.232+0000 7fa046cfc700 1 -- 172.21.15.53:0/996831859 wait complete. 2024-01-21T19:59:13.531 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:13 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:59:12.105007+0000 mgr.smithi053.fgsmol (mgr.14164) 4 : audit [DBG] from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T19:59:13.531 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:13 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:59:12.349221+0000 mon.smithi053 (mon.0) 107 : audit [INF] from='mgr.14164 172.21.15.53:0/2331671680' entity='mgr.smithi053.fgsmol' 2024-01-21T19:59:13.531 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:13 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:59:13.231506+0000 mon.smithi053 (mon.0) 108 : audit [DBG] from='client.? 172.21.15.53:0/996831859' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-01-21T19:59:13.596 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-21T19:59:13.604 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: disabled 2024-01-21T19:59:13.605 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl is-active firewalld.service 2024-01-21T19:59:13.612 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: inactive 2024-01-21T19:59:13.613 INFO:teuthology.orchestra.run.smithi053.stderr:firewalld.service is not enabled 2024-01-21T19:59:13.613 INFO:teuthology.orchestra.run.smithi053.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-01-21T19:59:13.615 INFO:teuthology.orchestra.run.smithi053.stderr:Ceph Dashboard is now available at: 2024-01-21T19:59:13.615 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-01-21T19:59:13.616 INFO:teuthology.orchestra.run.smithi053.stderr: URL: https://smithi053.front.sepia.ceph.com:8443/ 2024-01-21T19:59:13.616 INFO:teuthology.orchestra.run.smithi053.stderr: User: admin 2024-01-21T19:59:13.616 INFO:teuthology.orchestra.run.smithi053.stderr: Password: sj8ebjhool 2024-01-21T19:59:13.616 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-01-21T19:59:13.617 INFO:teuthology.orchestra.run.smithi053.stderr:You can access the Ceph CLI with: 2024-01-21T19:59:13.617 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-01-21T19:59:13.617 INFO:teuthology.orchestra.run.smithi053.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-01-21T19:59:13.617 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-01-21T19:59:13.617 INFO:teuthology.orchestra.run.smithi053.stderr:Please consider enabling telemetry to help improve Ceph: 2024-01-21T19:59:13.617 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-01-21T19:59:13.617 INFO:teuthology.orchestra.run.smithi053.stderr: ceph telemetry on 2024-01-21T19:59:13.617 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-01-21T19:59:13.617 INFO:teuthology.orchestra.run.smithi053.stderr:For more information see: 2024-01-21T19:59:13.617 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-01-21T19:59:13.617 INFO:teuthology.orchestra.run.smithi053.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2024-01-21T19:59:13.617 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-01-21T19:59:13.618 INFO:teuthology.orchestra.run.smithi053.stderr:Bootstrap complete. 2024-01-21T19:59:13.618 INFO:teuthology.orchestra.run.smithi053.stderr:Releasing lock 140640071911912 on /run/cephadm/57f6722c-b897-11ee-95b0-87774f69a715.lock 2024-01-21T19:59:13.618 INFO:teuthology.orchestra.run.smithi053.stderr:Lock 140640071911912 released on /run/cephadm/57f6722c-b897-11ee-95b0-87774f69a715.lock 2024-01-21T19:59:13.650 INFO:tasks.cephadm:Fetching config... 2024-01-21T19:59:13.650 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-21T19:59:13.650 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-01-21T19:59:13.709 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-01-21T19:59:13.709 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-21T19:59:13.709 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-01-21T19:59:13.767 INFO:tasks.cephadm:Fetching mon keyring... 2024-01-21T19:59:13.768 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-21T19:59:13.768 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/57f6722c-b897-11ee-95b0-87774f69a715/mon.smithi053/keyring of=/dev/stdout 2024-01-21T19:59:13.835 INFO:tasks.cephadm:Fetching pub ssh key... 2024-01-21T19:59:13.835 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-21T19:59:13.835 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-01-21T19:59:13.893 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-01-21T19:59:13.894 DEBUG:teuthology.orchestra.run.smithi053:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDIeCV1knv5cd28RNucCPaXPz2KDU4SO6DXjBNlZ1zrk6ikhUc9O/ds7lVcsVom+6z9+NKYnI1K6NrcFC/3mG/F0GEef3/iWlz9NtdLBSA/LODEH+GPnx67D7c6E0e0T+gXEqR/AdUrgjHpqvBlU6yfna+LnUPOaFtPsuUiWn57N/WmPCUJiPkysngopHHsYYiEYVwSTVg1pGbURUGnTJSi1LOBBpgQH05IJR5swYg2QuHuQS5U1X0GmnepSNatC3VozgaxpwfrU0O3h8copBEmOvQS/aIyVilW4EYTJNBQLc+ZD8Va44wbgpstFCOuLKrPHrOBIPUFcrcel960HYECFjjjVqtWEl7saQQI459YvvK0dqBMVXvpI1ZCvfj7rAQXt1qunz6UY8tT+9lrCqL5UE+49zzv9/CpI6W6Bh2eMMkc1ThJxOP62MldS3272FlaO5e1x1BA8cZy3LypxppHv25jiQvRyYkfLfq77a9VE45gvcO8rzTc/he3EXYq7wU= ceph-57f6722c-b897-11ee-95b0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-21T19:59:13.970 INFO:teuthology.orchestra.run.smithi053.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDIeCV1knv5cd28RNucCPaXPz2KDU4SO6DXjBNlZ1zrk6ikhUc9O/ds7lVcsVom+6z9+NKYnI1K6NrcFC/3mG/F0GEef3/iWlz9NtdLBSA/LODEH+GPnx67D7c6E0e0T+gXEqR/AdUrgjHpqvBlU6yfna+LnUPOaFtPsuUiWn57N/WmPCUJiPkysngopHHsYYiEYVwSTVg1pGbURUGnTJSi1LOBBpgQH05IJR5swYg2QuHuQS5U1X0GmnepSNatC3VozgaxpwfrU0O3h8copBEmOvQS/aIyVilW4EYTJNBQLc+ZD8Va44wbgpstFCOuLKrPHrOBIPUFcrcel960HYECFjjjVqtWEl7saQQI459YvvK0dqBMVXvpI1ZCvfj7rAQXt1qunz6UY8tT+9lrCqL5UE+49zzv9/CpI6W6Bh2eMMkc1ThJxOP62MldS3272FlaO5e1x1BA8cZy3LypxppHv25jiQvRyYkfLfq77a9VE45gvcO8rzTc/he3EXYq7wU= ceph-57f6722c-b897-11ee-95b0-87774f69a715 2024-01-21T19:59:13.981 DEBUG:teuthology.orchestra.run.smithi146:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDIeCV1knv5cd28RNucCPaXPz2KDU4SO6DXjBNlZ1zrk6ikhUc9O/ds7lVcsVom+6z9+NKYnI1K6NrcFC/3mG/F0GEef3/iWlz9NtdLBSA/LODEH+GPnx67D7c6E0e0T+gXEqR/AdUrgjHpqvBlU6yfna+LnUPOaFtPsuUiWn57N/WmPCUJiPkysngopHHsYYiEYVwSTVg1pGbURUGnTJSi1LOBBpgQH05IJR5swYg2QuHuQS5U1X0GmnepSNatC3VozgaxpwfrU0O3h8copBEmOvQS/aIyVilW4EYTJNBQLc+ZD8Va44wbgpstFCOuLKrPHrOBIPUFcrcel960HYECFjjjVqtWEl7saQQI459YvvK0dqBMVXvpI1ZCvfj7rAQXt1qunz6UY8tT+9lrCqL5UE+49zzv9/CpI6W6Bh2eMMkc1ThJxOP62MldS3272FlaO5e1x1BA8cZy3LypxppHv25jiQvRyYkfLfq77a9VE45gvcO8rzTc/he3EXYq7wU= ceph-57f6722c-b897-11ee-95b0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-21T19:59:14.026 INFO:teuthology.orchestra.run.smithi146.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDIeCV1knv5cd28RNucCPaXPz2KDU4SO6DXjBNlZ1zrk6ikhUc9O/ds7lVcsVom+6z9+NKYnI1K6NrcFC/3mG/F0GEef3/iWlz9NtdLBSA/LODEH+GPnx67D7c6E0e0T+gXEqR/AdUrgjHpqvBlU6yfna+LnUPOaFtPsuUiWn57N/WmPCUJiPkysngopHHsYYiEYVwSTVg1pGbURUGnTJSi1LOBBpgQH05IJR5swYg2QuHuQS5U1X0GmnepSNatC3VozgaxpwfrU0O3h8copBEmOvQS/aIyVilW4EYTJNBQLc+ZD8Va44wbgpstFCOuLKrPHrOBIPUFcrcel960HYECFjjjVqtWEl7saQQI459YvvK0dqBMVXvpI1ZCvfj7rAQXt1qunz6UY8tT+9lrCqL5UE+49zzv9/CpI6W6Bh2eMMkc1ThJxOP62MldS3272FlaO5e1x1BA8cZy3LypxppHv25jiQvRyYkfLfq77a9VE45gvcO8rzTc/he3EXYq7wU= ceph-57f6722c-b897-11ee-95b0-87774f69a715 2024-01-21T19:59:14.039 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-01-21T19:59:15.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:15.215+0000 7feebb8ea700 1 -- 172.21.15.53:0/2319046845 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feeb40f4740 msgr2=0x7feeb40f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:15.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:15.215+0000 7feebb8ea700 1 --2- 172.21.15.53:0/2319046845 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feeb40f4740 0x7feeb40f4b60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7feeb0009fb0 tx=0x7feeb0009b30).stop 2024-01-21T19:59:15.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:15.215+0000 7feebb8ea700 1 -- 172.21.15.53:0/2319046845 shutdown_connections 2024-01-21T19:59:15.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:15.215+0000 7feebb8ea700 1 --2- 172.21.15.53:0/2319046845 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feeb40f4740 0x7feeb40f4b60 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:15.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:15.215+0000 7feebb8ea700 1 -- 172.21.15.53:0/2319046845 >> 172.21.15.53:0/2319046845 conn(0x7feeb40f0260 msgr2=0x7feeb40f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:15.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:15.215+0000 7feebb8ea700 1 -- 172.21.15.53:0/2319046845 shutdown_connections 2024-01-21T19:59:15.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:15.215+0000 7feebb8ea700 1 -- 172.21.15.53:0/2319046845 wait complete. 2024-01-21T19:59:15.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:15.216+0000 7feebb8ea700 1 Processor -- start 2024-01-21T19:59:15.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:15.216+0000 7feebb8ea700 1 -- start start 2024-01-21T19:59:15.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:15.216+0000 7feebb8ea700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feeb40f4740 0x7feeb41066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:15.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:15.216+0000 7feebb8ea700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeb4106bd0 con 0x7feeb40f4740 2024-01-21T19:59:15.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:15.216+0000 7feeb9686700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feeb40f4740 0x7feeb41066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:15.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:15.216+0000 7feeb9686700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feeb40f4740 0x7feeb41066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:40118/0 (socket says 172.21.15.53:40118) 2024-01-21T19:59:15.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:15.216+0000 7feeb9686700 1 -- 172.21.15.53:0/504195483 learned_addr learned my addr 172.21.15.53:0/504195483 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:59:15.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:15.216+0000 7feeb9686700 1 -- 172.21.15.53:0/504195483 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeb0005040 con 0x7feeb40f4740 2024-01-21T19:59:15.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:15.216+0000 7feeb9686700 1 --2- 172.21.15.53:0/504195483 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feeb40f4740 0x7feeb41066e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7feeb0004dd0 tx=0x7feeb000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:15.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:15.216+0000 7feea5ffb700 1 -- 172.21.15.53:0/504195483 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feeb0009320 con 0x7feeb40f4740 2024-01-21T19:59:15.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:15.216+0000 7feea5ffb700 1 -- 172.21.15.53:0/504195483 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7feeb0013680 con 0x7feeb40f4740 2024-01-21T19:59:15.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:15.216+0000 7feebb8ea700 1 -- 172.21.15.53:0/504195483 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feeb4106dd0 con 0x7feeb40f4740 2024-01-21T19:59:15.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:15.216+0000 7feea5ffb700 1 -- 172.21.15.53:0/504195483 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feeb001ecb0 con 0x7feeb40f4740 2024-01-21T19:59:15.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:15.217+0000 7feebb8ea700 1 -- 172.21.15.53:0/504195483 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feeb4107260 con 0x7feeb40f4740 2024-01-21T19:59:15.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:15.217+0000 7feea5ffb700 1 -- 172.21.15.53:0/504195483 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feeb001e420 con 0x7feeb40f4740 2024-01-21T19:59:15.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:15.218+0000 7feea5ffb700 1 --2- 172.21.15.53:0/504195483 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7feea002ff60 0x7feea0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:15.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:15.218+0000 7feea5ffb700 1 -- 172.21.15.53:0/504195483 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feeb0040df0 con 0x7feeb40f4740 2024-01-21T19:59:15.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:15.218+0000 7feeb8e85700 1 --2- 172.21.15.53:0/504195483 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7feea002ff60 0x7feea0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:15.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:15.218+0000 7feebb8ea700 1 -- 172.21.15.53:0/504195483 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feeb4059100 con 0x7feeb40f4740 2024-01-21T19:59:15.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:15.218+0000 7feeb8e85700 1 --2- 172.21.15.53:0/504195483 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7feea002ff60 0x7feea0032420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7feea80096a0 tx=0x7feea8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:15.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:15.222+0000 7feea5ffb700 1 -- 172.21.15.53:0/504195483 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feeb00095f0 con 0x7feeb40f4740 2024-01-21T19:59:15.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:15.384+0000 7feebb8ea700 1 -- 172.21.15.53:0/504195483 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7feeb4045ad0 con 0x7feeb40f4740 2024-01-21T19:59:15.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:15.387+0000 7feea5ffb700 1 -- 172.21.15.53:0/504195483 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v10)=0 v10) v1 ==== 125+0+0 (secure 0 0 0) 0x7feeb0004490 con 0x7feeb40f4740 2024-01-21T19:59:15.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:15.389+0000 7feebb8ea700 1 -- 172.21.15.53:0/504195483 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7feea002ff60 msgr2=0x7feea0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:15.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:15.389+0000 7feebb8ea700 1 --2- 172.21.15.53:0/504195483 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7feea002ff60 0x7feea0032420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7feea80096a0 tx=0x7feea8006b40).stop 2024-01-21T19:59:15.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:15.389+0000 7feebb8ea700 1 -- 172.21.15.53:0/504195483 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feeb40f4740 msgr2=0x7feeb41066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:15.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:15.389+0000 7feebb8ea700 1 --2- 172.21.15.53:0/504195483 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feeb40f4740 0x7feeb41066e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7feeb0004dd0 tx=0x7feeb000b040).stop 2024-01-21T19:59:15.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:15.389+0000 7feebb8ea700 1 -- 172.21.15.53:0/504195483 shutdown_connections 2024-01-21T19:59:15.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:15.389+0000 7feebb8ea700 1 --2- 172.21.15.53:0/504195483 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7feea002ff60 0x7feea0032420 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:15.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:15.389+0000 7feebb8ea700 1 --2- 172.21.15.53:0/504195483 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feeb40f4740 0x7feeb41066e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:15.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:15.389+0000 7feebb8ea700 1 -- 172.21.15.53:0/504195483 >> 172.21.15.53:0/504195483 conn(0x7feeb40f0260 msgr2=0x7feeb40f0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:15.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:15.389+0000 7feebb8ea700 1 -- 172.21.15.53:0/504195483 shutdown_connections 2024-01-21T19:59:15.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:15.389+0000 7feebb8ea700 1 -- 172.21.15.53:0/504195483 wait complete. 2024-01-21T19:59:15.969 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi146 2024-01-21T19:59:15.970 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-21T19:59:15.970 DEBUG:teuthology.orchestra.run.smithi146:> dd of=/etc/ceph/ceph.conf 2024-01-21T19:59:15.991 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-21T19:59:15.991 DEBUG:teuthology.orchestra.run.smithi146:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-21T19:59:16.052 INFO:tasks.cephadm:Adding host smithi146 to orchestrator... 2024-01-21T19:59:16.052 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph orch host add smithi146 2024-01-21T19:59:16.434 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:59:15.147760+0000 mon.smithi053 (mon.0) 109 : cluster [DBG] mgrmap e17: smithi053.fgsmol(active, since 6s) 2024-01-21T19:59:16.434 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:59:15.388701+0000 mon.smithi053 (mon.0) 110 : audit [INF] from='client.? 172.21.15.53:0/504195483' entity='client.admin' 2024-01-21T19:59:18.434 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:18 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:59:17.160750+0000 mon.smithi053 (mon.0) 111 : audit [INF] from='mgr.14164 172.21.15.53:0/2331671680' entity='mgr.smithi053.fgsmol' 2024-01-21T19:59:18.434 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:18 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:59:17.387485+0000 mon.smithi053 (mon.0) 112 : audit [INF] from='mgr.14164 172.21.15.53:0/2331671680' entity='mgr.smithi053.fgsmol' 2024-01-21T19:59:18.434 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:18 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:59:18.052862+0000 mon.smithi053 (mon.0) 113 : audit [INF] from='mgr.14164 172.21.15.53:0/2331671680' entity='mgr.smithi053.fgsmol' 2024-01-21T19:59:21.684 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:21 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:59:20.407931+0000 mon.smithi053 (mon.0) 114 : audit [INF] from='mgr.14164 172.21.15.53:0/2331671680' entity='mgr.smithi053.fgsmol' 2024-01-21T19:59:21.684 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:21 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:59:20.685207+0000 mon.smithi053 (mon.0) 115 : audit [INF] from='mgr.14164 172.21.15.53:0/2331671680' entity='mgr.smithi053.fgsmol' 2024-01-21T19:59:21.684 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:21 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:59:20.687209+0000 mon.smithi053 (mon.0) 116 : audit [INF] from='mgr.14164 172.21.15.53:0/2331671680' entity='mgr.smithi053.fgsmol' 2024-01-21T19:59:21.684 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:21 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cephadm 2024-01-21T19:59:20.693709+0000 mgr.smithi053.fgsmol (mgr.14164) 5 : cephadm [INF] Deploying daemon alertmanager.smithi053 on smithi053 2024-01-21T19:59:30.434 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:30 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:59:29.126307+0000 mgr.smithi053.fgsmol (mgr.14164) 6 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:59:32.684 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:32 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:59:31.126612+0000 mgr.smithi053.fgsmol (mgr.14164) 7 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:59:34.684 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:34 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:59:33.126882+0000 mgr.smithi053.fgsmol (mgr.14164) 8 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:59:36.684 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:36 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:59:35.127173+0000 mgr.smithi053.fgsmol (mgr.14164) 9 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:59:36.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:36.954+0000 7f23c29da700 1 -- 172.21.15.146:0/3615500106 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f23bc0f6510 msgr2=0x7f23bc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:36.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:36.954+0000 7f23c29da700 1 --2- 172.21.15.146:0/3615500106 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f23bc0f6510 0x7f23bc0f6930 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f23ac009fb0 tx=0x7f23ac009b30).stop 2024-01-21T19:59:36.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:36.954+0000 7f23c29da700 1 -- 172.21.15.146:0/3615500106 shutdown_connections 2024-01-21T19:59:36.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:36.954+0000 7f23c29da700 1 --2- 172.21.15.146:0/3615500106 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f23bc0f6510 0x7f23bc0f6930 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:36.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:36.954+0000 7f23c29da700 1 -- 172.21.15.146:0/3615500106 >> 172.21.15.146:0/3615500106 conn(0x7f23bc0f1c70 msgr2=0x7f23bc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:36.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:36.954+0000 7f23c29da700 1 -- 172.21.15.146:0/3615500106 shutdown_connections 2024-01-21T19:59:36.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:36.954+0000 7f23c29da700 1 -- 172.21.15.146:0/3615500106 wait complete. 2024-01-21T19:59:36.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:36.954+0000 7f23c29da700 1 Processor -- start 2024-01-21T19:59:36.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:36.954+0000 7f23c29da700 1 -- start start 2024-01-21T19:59:36.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:36.955+0000 7f23c29da700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f23bc0f6510 0x7f23bc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:36.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:36.955+0000 7f23c29da700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23bc106c20 con 0x7f23bc0f6510 2024-01-21T19:59:36.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:36.955+0000 7f23bbfff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f23bc0f6510 0x7f23bc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:36.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:36.955+0000 7f23bbfff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f23bc0f6510 0x7f23bc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:51920/0 (socket says 172.21.15.146:51920) 2024-01-21T19:59:36.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:36.955+0000 7f23bbfff700 1 -- 172.21.15.146:0/3152260108 learned_addr learned my addr 172.21.15.146:0/3152260108 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T19:59:36.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:36.955+0000 7f23bbfff700 1 -- 172.21.15.146:0/3152260108 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23ac005040 con 0x7f23bc0f6510 2024-01-21T19:59:36.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:36.955+0000 7f23bbfff700 1 --2- 172.21.15.146:0/3152260108 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f23bc0f6510 0x7f23bc106730 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f23ac013510 tx=0x7f23ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:36.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:36.955+0000 7f23b8ff9700 1 -- 172.21.15.146:0/3152260108 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23ac009320 con 0x7f23bc0f6510 2024-01-21T19:59:36.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:36.955+0000 7f23c29da700 1 -- 172.21.15.146:0/3152260108 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23bc106e20 con 0x7f23bc0f6510 2024-01-21T19:59:36.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:36.955+0000 7f23b8ff9700 1 -- 172.21.15.146:0/3152260108 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f23ac013f00 con 0x7f23bc0f6510 2024-01-21T19:59:36.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:36.955+0000 7f23b8ff9700 1 -- 172.21.15.146:0/3152260108 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23ac014d90 con 0x7f23bc0f6510 2024-01-21T19:59:36.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:36.955+0000 7f23c29da700 1 -- 172.21.15.146:0/3152260108 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23bc1072b0 con 0x7f23bc0f6510 2024-01-21T19:59:36.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:36.956+0000 7f23b8ff9700 1 -- 172.21.15.146:0/3152260108 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f23ac0095f0 con 0x7f23bc0f6510 2024-01-21T19:59:36.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:36.956+0000 7f23c29da700 1 -- 172.21.15.146:0/3152260108 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23bc045ad0 con 0x7f23bc0f6510 2024-01-21T19:59:36.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:36.956+0000 7f23b8ff9700 1 --2- 172.21.15.146:0/3152260108 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f23a402ff60 0x7f23a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:36.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:36.956+0000 7f23b8ff9700 1 -- 172.21.15.146:0/3152260108 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f23ac041000 con 0x7f23bc0f6510 2024-01-21T19:59:36.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:36.957+0000 7f23bb7fe700 1 --2- 172.21.15.146:0/3152260108 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f23a402ff60 0x7f23a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:36.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:36.957+0000 7f23bb7fe700 1 --2- 172.21.15.146:0/3152260108 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f23a402ff60 0x7f23a4032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f23b00096a0 tx=0x7f23b0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:36.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:36.961+0000 7f23b8ff9700 1 -- 172.21.15.146:0/3152260108 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f23ac014420 con 0x7f23bc0f6510 2024-01-21T19:59:37.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:37.131+0000 7f23c29da700 1 -- 172.21.15.146:0/3152260108 --> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi146", "target": ["mon-mgr", ""]}) v1 -- 0x7f23bc0f2e70 con 0x7f23a402ff60 2024-01-21T19:59:37.714 INFO:teuthology.orchestra.run.smithi146.stdout:Added host 'smithi146' 2024-01-21T19:59:37.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:37.721+0000 7f23b8ff9700 1 -- 172.21.15.146:0/3152260108 <== mgr.14164 v2:172.21.15.53:6800/3203971228 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f23bc0f2e70 con 0x7f23a402ff60 2024-01-21T19:59:37.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:37.723+0000 7f23c29da700 1 -- 172.21.15.146:0/3152260108 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f23a402ff60 msgr2=0x7f23a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:37.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:37.723+0000 7f23c29da700 1 --2- 172.21.15.146:0/3152260108 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f23a402ff60 0x7f23a4032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f23b00096a0 tx=0x7f23b0006b40).stop 2024-01-21T19:59:37.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:37.723+0000 7f23c29da700 1 -- 172.21.15.146:0/3152260108 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f23bc0f6510 msgr2=0x7f23bc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:37.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:37.723+0000 7f23c29da700 1 --2- 172.21.15.146:0/3152260108 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f23bc0f6510 0x7f23bc106730 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f23ac013510 tx=0x7f23ac00b040).stop 2024-01-21T19:59:37.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:37.723+0000 7f23c29da700 1 -- 172.21.15.146:0/3152260108 shutdown_connections 2024-01-21T19:59:37.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:37.723+0000 7f23c29da700 1 --2- 172.21.15.146:0/3152260108 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f23a402ff60 0x7f23a4032420 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:37.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:37.723+0000 7f23c29da700 1 --2- 172.21.15.146:0/3152260108 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f23bc0f6510 0x7f23bc106730 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:37.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:37.723+0000 7f23c29da700 1 -- 172.21.15.146:0/3152260108 >> 172.21.15.146:0/3152260108 conn(0x7f23bc0f1c70 msgr2=0x7f23bc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:37.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:37.724+0000 7f23c29da700 1 -- 172.21.15.146:0/3152260108 shutdown_connections 2024-01-21T19:59:37.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:37.724+0000 7f23c29da700 1 -- 172.21.15.146:0/3152260108 wait complete. 2024-01-21T19:59:38.084 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph orch host ls --format=json 2024-01-21T19:59:39.184 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:38 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:59:37.127520+0000 mgr.smithi053.fgsmol (mgr.14164) 10 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:59:39.184 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:38 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:59:37.133728+0000 mgr.smithi053.fgsmol (mgr.14164) 11 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi146", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T19:59:39.184 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:38 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:59:37.721424+0000 mon.smithi053 (mon.0) 117 : audit [INF] from='mgr.14164 172.21.15.53:0/2331671680' entity='mgr.smithi053.fgsmol' 2024-01-21T19:59:39.184 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:38 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cephadm 2024-01-21T19:59:37.721863+0000 mgr.smithi053.fgsmol (mgr.14164) 12 : cephadm [INF] Added host smithi146 2024-01-21T19:59:39.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:39.310+0000 7f8dd3c21700 1 -- 172.21.15.146:0/4214448586 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8dcc0f6a10 msgr2=0x7f8dcc0f6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:39.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:39.310+0000 7f8dd3c21700 1 --2- 172.21.15.146:0/4214448586 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8dcc0f6a10 0x7f8dcc0f6e30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f8dc0009fb0 tx=0x7f8dc0009b30).stop 2024-01-21T19:59:39.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:39.310+0000 7f8dd3c21700 1 -- 172.21.15.146:0/4214448586 shutdown_connections 2024-01-21T19:59:39.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:39.310+0000 7f8dd3c21700 1 --2- 172.21.15.146:0/4214448586 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8dcc0f6a10 0x7f8dcc0f6e30 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:39.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:39.310+0000 7f8dd3c21700 1 -- 172.21.15.146:0/4214448586 >> 172.21.15.146:0/4214448586 conn(0x7f8dcc0f02e0 msgr2=0x7f8dcc0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:39.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:39.310+0000 7f8dd3c21700 1 -- 172.21.15.146:0/4214448586 shutdown_connections 2024-01-21T19:59:39.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:39.310+0000 7f8dd3c21700 1 -- 172.21.15.146:0/4214448586 wait complete. 2024-01-21T19:59:39.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:39.311+0000 7f8dd3c21700 1 Processor -- start 2024-01-21T19:59:39.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:39.311+0000 7f8dd3c21700 1 -- start start 2024-01-21T19:59:39.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:39.311+0000 7f8dd3c21700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8dcc0f6a10 0x7f8dcc10b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:39.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:39.311+0000 7f8dd3c21700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dcc1071e0 con 0x7f8dcc0f6a10 2024-01-21T19:59:39.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:39.311+0000 7f8dd19bd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8dcc0f6a10 0x7f8dcc10b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:39.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:39.311+0000 7f8dd19bd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8dcc0f6a10 0x7f8dcc10b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:38218/0 (socket says 172.21.15.146:38218) 2024-01-21T19:59:39.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:39.311+0000 7f8dd19bd700 1 -- 172.21.15.146:0/1356497421 learned_addr learned my addr 172.21.15.146:0/1356497421 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T19:59:39.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:39.312+0000 7f8dd19bd700 1 -- 172.21.15.146:0/1356497421 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc0005040 con 0x7f8dcc0f6a10 2024-01-21T19:59:39.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:39.312+0000 7f8dd19bd700 1 --2- 172.21.15.146:0/1356497421 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8dcc0f6a10 0x7f8dcc10b7f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f8dc0013510 tx=0x7f8dc000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:39.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:39.312+0000 7f8dbe7fc700 1 -- 172.21.15.146:0/1356497421 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8dc0009320 con 0x7f8dcc0f6a10 2024-01-21T19:59:39.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:39.312+0000 7f8dd3c21700 1 -- 172.21.15.146:0/1356497421 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dcc0f5330 con 0x7f8dcc0f6a10 2024-01-21T19:59:39.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:39.312+0000 7f8dbe7fc700 1 -- 172.21.15.146:0/1356497421 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8dc0013f00 con 0x7f8dcc0f6a10 2024-01-21T19:59:39.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:39.312+0000 7f8dbe7fc700 1 -- 172.21.15.146:0/1356497421 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8dc0014d90 con 0x7f8dcc0f6a10 2024-01-21T19:59:39.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:39.312+0000 7f8dd3c21700 1 -- 172.21.15.146:0/1356497421 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dcc10bce0 con 0x7f8dcc0f6a10 2024-01-21T19:59:39.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:39.313+0000 7f8dbe7fc700 1 -- 172.21.15.146:0/1356497421 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8dc0014420 con 0x7f8dcc0f6a10 2024-01-21T19:59:39.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:39.313+0000 7f8dbe7fc700 1 --2- 172.21.15.146:0/1356497421 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f8db802ff60 0x7f8db8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:39.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:39.313+0000 7f8dd3c21700 1 -- 172.21.15.146:0/1356497421 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dcc045ad0 con 0x7f8dcc0f6a10 2024-01-21T19:59:39.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:39.313+0000 7f8dbe7fc700 1 -- 172.21.15.146:0/1356497421 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8dc0040fb0 con 0x7f8dcc0f6a10 2024-01-21T19:59:39.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:39.313+0000 7f8dd11bc700 1 --2- 172.21.15.146:0/1356497421 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f8db802ff60 0x7f8db8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:39.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:39.314+0000 7f8dd11bc700 1 --2- 172.21.15.146:0/1356497421 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f8db802ff60 0x7f8db8032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f8dc80096a0 tx=0x7f8dc8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:39.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:39.318+0000 7f8dbe7fc700 1 -- 172.21.15.146:0/1356497421 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8dc001dd50 con 0x7f8dcc0f6a10 2024-01-21T19:59:39.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:39.487+0000 7f8dd3c21700 1 -- 172.21.15.146:0/1356497421 --> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8dcc0f14b0 con 0x7f8db802ff60 2024-01-21T19:59:39.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:39.489+0000 7f8dbe7fc700 1 -- 172.21.15.146:0/1356497421 <== mgr.14164 v2:172.21.15.53:6800/3203971228 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+156 (secure 0 0 0) 0x7f8dcc0f14b0 con 0x7f8db802ff60 2024-01-21T19:59:39.480 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:59:39.481 INFO:teuthology.orchestra.run.smithi146.stdout:[{"addr": "172.21.15.53", "hostname": "smithi053", "labels": [], "status": ""}, {"addr": "smithi146", "hostname": "smithi146", "labels": [], "status": ""}] 2024-01-21T19:59:39.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:39.490+0000 7f8dd3c21700 1 -- 172.21.15.146:0/1356497421 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f8db802ff60 msgr2=0x7f8db8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:39.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:39.490+0000 7f8dd3c21700 1 --2- 172.21.15.146:0/1356497421 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f8db802ff60 0x7f8db8032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f8dc80096a0 tx=0x7f8dc8006b40).stop 2024-01-21T19:59:39.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:39.490+0000 7f8dd3c21700 1 -- 172.21.15.146:0/1356497421 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8dcc0f6a10 msgr2=0x7f8dcc10b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:39.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:39.490+0000 7f8dd3c21700 1 --2- 172.21.15.146:0/1356497421 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8dcc0f6a10 0x7f8dcc10b7f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f8dc0013510 tx=0x7f8dc000b040).stop 2024-01-21T19:59:39.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:39.490+0000 7f8dd3c21700 1 -- 172.21.15.146:0/1356497421 shutdown_connections 2024-01-21T19:59:39.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:39.490+0000 7f8dd3c21700 1 --2- 172.21.15.146:0/1356497421 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f8db802ff60 0x7f8db8032420 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:39.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:39.490+0000 7f8dd3c21700 1 --2- 172.21.15.146:0/1356497421 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8dcc0f6a10 0x7f8dcc10b7f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:39.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:39.490+0000 7f8dd3c21700 1 -- 172.21.15.146:0/1356497421 >> 172.21.15.146:0/1356497421 conn(0x7f8dcc0f02e0 msgr2=0x7f8dcc0f0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:39.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:39.490+0000 7f8dd3c21700 1 -- 172.21.15.146:0/1356497421 shutdown_connections 2024-01-21T19:59:39.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:39.490+0000 7f8dd3c21700 1 -- 172.21.15.146:0/1356497421 wait complete. 2024-01-21T19:59:39.861 INFO:tasks.cephadm:Setting crush tunables to default 2024-01-21T19:59:39.862 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph osd crush tunables default 2024-01-21T19:59:40.860 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:40 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:59:39.127764+0000 mgr.smithi053.fgsmol (mgr.14164) 13 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:59:40.860 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:40 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:59:39.489787+0000 mgr.smithi053.fgsmol (mgr.14164) 14 : audit [DBG] from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-21T19:59:41.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:41.052+0000 7f92bc958700 1 -- 172.21.15.53:0/382980063 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92b80f51a0 msgr2=0x7f92b80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:41.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:41.052+0000 7f92bc958700 1 --2- 172.21.15.53:0/382980063 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92b80f51a0 0x7f92b80f55c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f92a8009fb0 tx=0x7f92a8009b30).stop 2024-01-21T19:59:41.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:41.052+0000 7f92bc958700 1 -- 172.21.15.53:0/382980063 shutdown_connections 2024-01-21T19:59:41.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:41.052+0000 7f92bc958700 1 --2- 172.21.15.53:0/382980063 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92b80f51a0 0x7f92b80f55c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:41.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:41.052+0000 7f92bc958700 1 -- 172.21.15.53:0/382980063 >> 172.21.15.53:0/382980063 conn(0x7f92b80f09a0 msgr2=0x7f92b80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:41.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:41.052+0000 7f92bc958700 1 -- 172.21.15.53:0/382980063 shutdown_connections 2024-01-21T19:59:41.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:41.052+0000 7f92bc958700 1 -- 172.21.15.53:0/382980063 wait complete. 2024-01-21T19:59:41.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:41.053+0000 7f92bc958700 1 Processor -- start 2024-01-21T19:59:41.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:41.053+0000 7f92bc958700 1 -- start start 2024-01-21T19:59:41.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:41.053+0000 7f92bc958700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92b80f51a0 0x7f92b80fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:41.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:41.053+0000 7f92bc958700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b80fb5a0 con 0x7f92b80f51a0 2024-01-21T19:59:41.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:41.053+0000 7f92b6d9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92b80f51a0 0x7f92b80fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:41.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:41.053+0000 7f92b6d9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92b80f51a0 0x7f92b80fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55322/0 (socket says 172.21.15.53:55322) 2024-01-21T19:59:41.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:41.053+0000 7f92b6d9d700 1 -- 172.21.15.53:0/1322165499 learned_addr learned my addr 172.21.15.53:0/1322165499 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-21T19:59:41.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:41.054+0000 7f92b6d9d700 1 -- 172.21.15.53:0/1322165499 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92a8005040 con 0x7f92b80f51a0 2024-01-21T19:59:41.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:41.054+0000 7f92b6d9d700 1 --2- 172.21.15.53:0/1322165499 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92b80f51a0 0x7f92b80fd120 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f92a8013510 tx=0x7f92a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:41.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:41.054+0000 7f929f7fe700 1 -- 172.21.15.53:0/1322165499 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f92a8009320 con 0x7f92b80f51a0 2024-01-21T19:59:41.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:41.054+0000 7f929f7fe700 1 -- 172.21.15.53:0/1322165499 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f92a8013f00 con 0x7f92b80f51a0 2024-01-21T19:59:41.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:41.054+0000 7f92bc958700 1 -- 172.21.15.53:0/1322165499 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92b80fb740 con 0x7f92b80f51a0 2024-01-21T19:59:41.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:41.054+0000 7f929f7fe700 1 -- 172.21.15.53:0/1322165499 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f92a8014dc0 con 0x7f92b80f51a0 2024-01-21T19:59:41.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:41.054+0000 7f92bc958700 1 -- 172.21.15.53:0/1322165499 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92b80fbbd0 con 0x7f92b80f51a0 2024-01-21T19:59:41.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:41.055+0000 7f929f7fe700 1 -- 172.21.15.53:0/1322165499 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f92a80263a0 con 0x7f92b80f51a0 2024-01-21T19:59:41.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:41.055+0000 7f929f7fe700 1 --2- 172.21.15.53:0/1322165499 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f92a002ffb0 0x7f92a0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:41.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:41.055+0000 7f929f7fe700 1 -- 172.21.15.53:0/1322165499 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f92a8041000 con 0x7f92b80f51a0 2024-01-21T19:59:41.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:41.055+0000 7f92bc958700 1 -- 172.21.15.53:0/1322165499 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92b8059100 con 0x7f92b80f51a0 2024-01-21T19:59:41.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:41.055+0000 7f92b659c700 1 --2- 172.21.15.53:0/1322165499 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f92a002ffb0 0x7f92a0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:41.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:41.055+0000 7f92b659c700 1 --2- 172.21.15.53:0/1322165499 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f92a002ffb0 0x7f92a0032470 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f92ac0096a0 tx=0x7f92ac006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:41.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:41.059+0000 7f929f7fe700 1 -- 172.21.15.53:0/1322165499 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f92a8012920 con 0x7f92b80f51a0 2024-01-21T19:59:41.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:41.225+0000 7f92bc958700 1 -- 172.21.15.53:0/1322165499 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f92b80fabe0 con 0x7f92b80f51a0 2024-01-21T19:59:41.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:41.724+0000 7f929f7fe700 1 -- 172.21.15.53:0/1322165499 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7f92a8012330 con 0x7f92b80f51a0 2024-01-21T19:59:41.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:41.726+0000 7f92bc958700 1 -- 172.21.15.53:0/1322165499 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f92a002ffb0 msgr2=0x7f92a0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:41.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:41.726+0000 7f92bc958700 1 --2- 172.21.15.53:0/1322165499 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f92a002ffb0 0x7f92a0032470 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f92ac0096a0 tx=0x7f92ac006b40).stop 2024-01-21T19:59:41.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:41.726+0000 7f92bc958700 1 -- 172.21.15.53:0/1322165499 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92b80f51a0 msgr2=0x7f92b80fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:41.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:41.726+0000 7f92bc958700 1 --2- 172.21.15.53:0/1322165499 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92b80f51a0 0x7f92b80fd120 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f92a8013510 tx=0x7f92a800b040).stop 2024-01-21T19:59:41.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:41.726+0000 7f92bc958700 1 -- 172.21.15.53:0/1322165499 shutdown_connections 2024-01-21T19:59:41.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:41.726+0000 7f92bc958700 1 --2- 172.21.15.53:0/1322165499 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f92a002ffb0 0x7f92a0032470 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:41.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:41.726+0000 7f92bc958700 1 --2- 172.21.15.53:0/1322165499 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92b80f51a0 0x7f92b80fd120 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:41.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:41.726+0000 7f92bc958700 1 -- 172.21.15.53:0/1322165499 >> 172.21.15.53:0/1322165499 conn(0x7f92b80f09a0 msgr2=0x7f92b805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:41.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:41.726+0000 7f92bc958700 1 -- 172.21.15.53:0/1322165499 shutdown_connections 2024-01-21T19:59:41.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-21T19:59:41.726+0000 7f92bc958700 1 -- 172.21.15.53:0/1322165499 wait complete. 2024-01-21T19:59:41.719 INFO:teuthology.orchestra.run.smithi053.stderr:adjusted tunables profile to default 2024-01-21T19:59:42.061 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:59:41.226768+0000 mon.smithi053 (mon.0) 118 : audit [INF] from='client.? 172.21.15.53:0/1322165499' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-01-21T19:59:42.124 INFO:tasks.cephadm:Adding mon.smithi053 on smithi053 2024-01-21T19:59:42.124 INFO:tasks.cephadm:Adding mon.smithi146 on smithi146 2024-01-21T19:59:42.124 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph orch apply mon '2;smithi053:172.21.15.53=smithi053;smithi146:172.21.15.146=smithi146' 2024-01-21T19:59:43.184 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:42 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:59:41.128028+0000 mgr.smithi053.fgsmol (mgr.14164) 15 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:59:43.184 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:42 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:59:41.725533+0000 mon.smithi053 (mon.0) 119 : audit [INF] from='client.? 172.21.15.53:0/1322165499' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-01-21T19:59:43.184 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:42 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:59:41.725591+0000 mon.smithi053 (mon.0) 120 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-01-21T19:59:43.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:43.305+0000 7f804a7c2700 1 -- 172.21.15.146:0/1759024458 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80440f5410 msgr2=0x7f80440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:43.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:43.305+0000 7f804a7c2700 1 --2- 172.21.15.146:0/1759024458 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80440f5410 0x7f80440f5830 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f8034009fb0 tx=0x7f8034009b30).stop 2024-01-21T19:59:43.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:43.305+0000 7f804a7c2700 1 -- 172.21.15.146:0/1759024458 shutdown_connections 2024-01-21T19:59:43.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:43.305+0000 7f804a7c2700 1 --2- 172.21.15.146:0/1759024458 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80440f5410 0x7f80440f5830 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:43.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:43.305+0000 7f804a7c2700 1 -- 172.21.15.146:0/1759024458 >> 172.21.15.146:0/1759024458 conn(0x7f80440f0b70 msgr2=0x7f80440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:43.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:43.305+0000 7f804a7c2700 1 -- 172.21.15.146:0/1759024458 shutdown_connections 2024-01-21T19:59:43.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:43.305+0000 7f804a7c2700 1 -- 172.21.15.146:0/1759024458 wait complete. 2024-01-21T19:59:43.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:43.305+0000 7f804a7c2700 1 Processor -- start 2024-01-21T19:59:43.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:43.306+0000 7f804a7c2700 1 -- start start 2024-01-21T19:59:43.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:43.306+0000 7f804a7c2700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80440f5410 0x7f8044106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:43.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:43.306+0000 7f804a7c2700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8044106c00 con 0x7f80440f5410 2024-01-21T19:59:43.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:43.306+0000 7f8043fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80440f5410 0x7f8044106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:43.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:43.306+0000 7f8043fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80440f5410 0x7f8044106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:38246/0 (socket says 172.21.15.146:38246) 2024-01-21T19:59:43.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:43.306+0000 7f8043fff700 1 -- 172.21.15.146:0/3653784332 learned_addr learned my addr 172.21.15.146:0/3653784332 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T19:59:43.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:43.306+0000 7f8043fff700 1 -- 172.21.15.146:0/3653784332 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8034005040 con 0x7f80440f5410 2024-01-21T19:59:43.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:43.307+0000 7f8043fff700 1 --2- 172.21.15.146:0/3653784332 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80440f5410 0x7f8044106710 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f8034013510 tx=0x7f803400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:43.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:43.307+0000 7f8040ff9700 1 -- 172.21.15.146:0/3653784332 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8034009320 con 0x7f80440f5410 2024-01-21T19:59:43.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:43.307+0000 7f804a7c2700 1 -- 172.21.15.146:0/3653784332 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8044106e00 con 0x7f80440f5410 2024-01-21T19:59:43.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:43.307+0000 7f8040ff9700 1 -- 172.21.15.146:0/3653784332 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8034013f00 con 0x7f80440f5410 2024-01-21T19:59:43.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:43.307+0000 7f804a7c2700 1 -- 172.21.15.146:0/3653784332 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8044107290 con 0x7f80440f5410 2024-01-21T19:59:43.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:43.307+0000 7f8040ff9700 1 -- 172.21.15.146:0/3653784332 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8034014d90 con 0x7f80440f5410 2024-01-21T19:59:43.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:43.308+0000 7f8040ff9700 1 -- 172.21.15.146:0/3653784332 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f80340095f0 con 0x7f80440f5410 2024-01-21T19:59:43.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:43.308+0000 7f804a7c2700 1 -- 172.21.15.146:0/3653784332 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8044045ad0 con 0x7f80440f5410 2024-01-21T19:59:43.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:43.308+0000 7f8040ff9700 1 --2- 172.21.15.146:0/3653784332 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f803002ff60 0x7f8030032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:43.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:43.308+0000 7f8040ff9700 1 -- 172.21.15.146:0/3653784332 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f80340410c0 con 0x7f80440f5410 2024-01-21T19:59:43.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:43.308+0000 7f80437fe700 1 --2- 172.21.15.146:0/3653784332 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f803002ff60 0x7f8030032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:43.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:43.308+0000 7f80437fe700 1 --2- 172.21.15.146:0/3653784332 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f803002ff60 0x7f8030032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f802c0096a0 tx=0x7f802c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:43.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:43.313+0000 7f8040ff9700 1 -- 172.21.15.146:0/3653784332 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8034014420 con 0x7f80440f5410 2024-01-21T19:59:43.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:43.490+0000 7f804a7c2700 1 -- 172.21.15.146:0/3653784332 --> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi053:172.21.15.53=smithi053;smithi146:172.21.15.146=smithi146", "target": ["mon-mgr", ""]}) v1 -- 0x7f80440f2aa0 con 0x7f803002ff60 2024-01-21T19:59:43.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:43.495+0000 7f8040ff9700 1 -- 172.21.15.146:0/3653784332 <== mgr.14164 v2:172.21.15.53:6800/3203971228 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f80440f2aa0 con 0x7f803002ff60 2024-01-21T19:59:43.487 INFO:teuthology.orchestra.run.smithi146.stdout:Scheduled mon update... 2024-01-21T19:59:43.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:43.497+0000 7f804a7c2700 1 -- 172.21.15.146:0/3653784332 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f803002ff60 msgr2=0x7f8030032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:43.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:43.497+0000 7f804a7c2700 1 --2- 172.21.15.146:0/3653784332 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f803002ff60 0x7f8030032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f802c0096a0 tx=0x7f802c006b40).stop 2024-01-21T19:59:43.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:43.497+0000 7f804a7c2700 1 -- 172.21.15.146:0/3653784332 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80440f5410 msgr2=0x7f8044106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:43.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:43.497+0000 7f804a7c2700 1 --2- 172.21.15.146:0/3653784332 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80440f5410 0x7f8044106710 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f8034013510 tx=0x7f803400b040).stop 2024-01-21T19:59:43.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:43.497+0000 7f804a7c2700 1 -- 172.21.15.146:0/3653784332 shutdown_connections 2024-01-21T19:59:43.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:43.497+0000 7f804a7c2700 1 --2- 172.21.15.146:0/3653784332 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f803002ff60 0x7f8030032420 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:43.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:43.497+0000 7f804a7c2700 1 --2- 172.21.15.146:0/3653784332 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80440f5410 0x7f8044106710 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:43.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:43.497+0000 7f804a7c2700 1 -- 172.21.15.146:0/3653784332 >> 172.21.15.146:0/3653784332 conn(0x7f80440f0b70 msgr2=0x7f804405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:43.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:43.497+0000 7f804a7c2700 1 -- 172.21.15.146:0/3653784332 shutdown_connections 2024-01-21T19:59:43.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:43.497+0000 7f804a7c2700 1 -- 172.21.15.146:0/3653784332 wait complete. 2024-01-21T19:59:43.920 DEBUG:teuthology.orchestra.run.smithi146:mon.smithi146> sudo journalctl -f -n 0 -u ceph-57f6722c-b897-11ee-95b0-87774f69a715@mon.smithi146.service 2024-01-21T19:59:43.922 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:59:43.922 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:59:43.951 INFO:journalctl@ceph.mon.smithi146.smithi146.stdout:-- Logs begin at Sun 2024-01-21 19:47:38 UTC. -- 2024-01-21T19:59:44.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:44 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:59:43.128433+0000 mgr.smithi053.fgsmol (mgr.14164) 16 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:59:44.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:44 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:59:43.492111+0000 mgr.smithi053.fgsmol (mgr.14164) 17 : audit [DBG] from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi053:172.21.15.53=smithi053;smithi146:172.21.15.146=smithi146", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T19:59:44.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:44 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cephadm 2024-01-21T19:59:43.493521+0000 mgr.smithi053.fgsmol (mgr.14164) 18 : cephadm [INF] Saving service mon spec with placement smithi053:172.21.15.53=smithi053;smithi146:172.21.15.146=smithi146;count:2 2024-01-21T19:59:44.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:44 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:59:43.496134+0000 mon.smithi053 (mon.0) 121 : audit [INF] from='mgr.14164 172.21.15.53:0/2331671680' entity='mgr.smithi053.fgsmol' 2024-01-21T19:59:45.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:45.117+0000 7f0315537700 1 -- 172.21.15.146:0/3311900524 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f03100f62a0 msgr2=0x7f03100f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:45.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:45.117+0000 7f0315537700 1 --2- 172.21.15.146:0/3311900524 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f03100f62a0 0x7f03100f66c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f0300009fb0 tx=0x7f0300009b30).stop 2024-01-21T19:59:45.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:45.117+0000 7f0315537700 1 -- 172.21.15.146:0/3311900524 shutdown_connections 2024-01-21T19:59:45.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:45.117+0000 7f0315537700 1 --2- 172.21.15.146:0/3311900524 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f03100f62a0 0x7f03100f66c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:45.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:45.117+0000 7f0315537700 1 -- 172.21.15.146:0/3311900524 >> 172.21.15.146:0/3311900524 conn(0x7f03100f1ac0 msgr2=0x7f03100f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:45.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:45.117+0000 7f0315537700 1 -- 172.21.15.146:0/3311900524 shutdown_connections 2024-01-21T19:59:45.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:45.117+0000 7f0315537700 1 -- 172.21.15.146:0/3311900524 wait complete. 2024-01-21T19:59:45.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:45.117+0000 7f0315537700 1 Processor -- start 2024-01-21T19:59:45.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:45.118+0000 7f0315537700 1 -- start start 2024-01-21T19:59:45.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:45.118+0000 7f0315537700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f03100f62a0 0x7f03101064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:45.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:45.118+0000 7f0315537700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0310106990 con 0x7f03100f62a0 2024-01-21T19:59:45.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:45.118+0000 7f030ffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f03100f62a0 0x7f03101064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:45.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:45.118+0000 7f030ffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f03100f62a0 0x7f03101064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:38260/0 (socket says 172.21.15.146:38260) 2024-01-21T19:59:45.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:45.118+0000 7f030ffff700 1 -- 172.21.15.146:0/55888228 learned_addr learned my addr 172.21.15.146:0/55888228 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T19:59:45.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:45.118+0000 7f030ffff700 1 -- 172.21.15.146:0/55888228 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0300005040 con 0x7f03100f62a0 2024-01-21T19:59:45.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:45.119+0000 7f030ffff700 1 --2- 172.21.15.146:0/55888228 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f03100f62a0 0x7f03101064a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f0300013510 tx=0x7f030000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:45.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:45.119+0000 7f030cff9700 1 -- 172.21.15.146:0/55888228 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0300009320 con 0x7f03100f62a0 2024-01-21T19:59:45.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:45.119+0000 7f0315537700 1 -- 172.21.15.146:0/55888228 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0310106b90 con 0x7f03100f62a0 2024-01-21T19:59:45.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:45.119+0000 7f030cff9700 1 -- 172.21.15.146:0/55888228 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0300013f00 con 0x7f03100f62a0 2024-01-21T19:59:45.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:45.119+0000 7f030cff9700 1 -- 172.21.15.146:0/55888228 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0300014dc0 con 0x7f03100f62a0 2024-01-21T19:59:45.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:45.119+0000 7f0315537700 1 -- 172.21.15.146:0/55888228 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0310107020 con 0x7f03100f62a0 2024-01-21T19:59:45.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:45.120+0000 7f030cff9700 1 -- 172.21.15.146:0/55888228 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0300014420 con 0x7f03100f62a0 2024-01-21T19:59:45.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:45.120+0000 7f0315537700 1 -- 172.21.15.146:0/55888228 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0310045ad0 con 0x7f03100f62a0 2024-01-21T19:59:45.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:45.120+0000 7f030cff9700 1 --2- 172.21.15.146:0/55888228 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f02f802ff60 0x7f02f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:45.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:45.120+0000 7f030cff9700 1 -- 172.21.15.146:0/55888228 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0300041110 con 0x7f03100f62a0 2024-01-21T19:59:45.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:45.120+0000 7f030f7fe700 1 --2- 172.21.15.146:0/55888228 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f02f802ff60 0x7f02f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:45.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:45.120+0000 7f030f7fe700 1 --2- 172.21.15.146:0/55888228 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f02f802ff60 0x7f02f8032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f03040096a0 tx=0x7f0304006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:45.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:45.124+0000 7f030cff9700 1 -- 172.21.15.146:0/55888228 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f03000095f0 con 0x7f03100f62a0 2024-01-21T19:59:45.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:45.330+0000 7f0315537700 1 -- 172.21.15.146:0/55888228 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0310107ed0 con 0x7f03100f62a0 2024-01-21T19:59:45.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:45.331+0000 7f030cff9700 1 -- 172.21.15.146:0/55888228 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f030001cb10 con 0x7f03100f62a0 2024-01-21T19:59:45.323 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:59:45.323 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:59:45.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:45.332+0000 7f0315537700 1 -- 172.21.15.146:0/55888228 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f02f802ff60 msgr2=0x7f02f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:45.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:45.332+0000 7f0315537700 1 --2- 172.21.15.146:0/55888228 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f02f802ff60 0x7f02f8032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f03040096a0 tx=0x7f0304006b40).stop 2024-01-21T19:59:45.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:45.332+0000 7f0315537700 1 -- 172.21.15.146:0/55888228 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f03100f62a0 msgr2=0x7f03101064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:45.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:45.332+0000 7f0315537700 1 --2- 172.21.15.146:0/55888228 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f03100f62a0 0x7f03101064a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f0300013510 tx=0x7f030000b040).stop 2024-01-21T19:59:45.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:45.333+0000 7f0315537700 1 -- 172.21.15.146:0/55888228 shutdown_connections 2024-01-21T19:59:45.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:45.333+0000 7f0315537700 1 --2- 172.21.15.146:0/55888228 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f02f802ff60 0x7f02f8032420 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:45.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:45.333+0000 7f0315537700 1 --2- 172.21.15.146:0/55888228 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f03100f62a0 0x7f03101064a0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:45.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:45.333+0000 7f0315537700 1 -- 172.21.15.146:0/55888228 >> 172.21.15.146:0/55888228 conn(0x7f03100f1ac0 msgr2=0x7f03100f2560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:45.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:45.333+0000 7f0315537700 1 -- 172.21.15.146:0/55888228 shutdown_connections 2024-01-21T19:59:45.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:45.333+0000 7f0315537700 1 -- 172.21.15.146:0/55888228 wait complete. 2024-01-21T19:59:45.325 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T19:59:45.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:45 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:59:45.332209+0000 mon.smithi053 (mon.0) 122 : audit [DBG] from='client.? 172.21.15.146:0/55888228' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:59:46.744 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:59:46.745 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:59:46.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:46 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:59:45.128663+0000 mgr.smithi053.fgsmol (mgr.14164) 19 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:59:48.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:48.094+0000 7f02b6ea3700 1 -- 172.21.15.146:0/1621277932 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02b00f6510 msgr2=0x7f02b00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:48.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:48.094+0000 7f02b6ea3700 1 --2- 172.21.15.146:0/1621277932 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02b00f6510 0x7f02b00f6930 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f02a4009fb0 tx=0x7f02a4009b30).stop 2024-01-21T19:59:48.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:48.094+0000 7f02b6ea3700 1 -- 172.21.15.146:0/1621277932 shutdown_connections 2024-01-21T19:59:48.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:48.094+0000 7f02b6ea3700 1 --2- 172.21.15.146:0/1621277932 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02b00f6510 0x7f02b00f6930 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:48.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:48.094+0000 7f02b6ea3700 1 -- 172.21.15.146:0/1621277932 >> 172.21.15.146:0/1621277932 conn(0x7f02b00f1c70 msgr2=0x7f02b00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:48.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:48.094+0000 7f02b6ea3700 1 -- 172.21.15.146:0/1621277932 shutdown_connections 2024-01-21T19:59:48.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:48.094+0000 7f02b6ea3700 1 -- 172.21.15.146:0/1621277932 wait complete. 2024-01-21T19:59:48.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:48.094+0000 7f02b6ea3700 1 Processor -- start 2024-01-21T19:59:48.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:48.094+0000 7f02b6ea3700 1 -- start start 2024-01-21T19:59:48.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:48.095+0000 7f02b6ea3700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02b00f6510 0x7f02b0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:48.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:48.095+0000 7f02b6ea3700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b0106c20 con 0x7f02b00f6510 2024-01-21T19:59:48.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:48.095+0000 7f02b4c3f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02b00f6510 0x7f02b0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:48.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:48.095+0000 7f02b4c3f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02b00f6510 0x7f02b0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:46460/0 (socket says 172.21.15.146:46460) 2024-01-21T19:59:48.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:48.095+0000 7f02b4c3f700 1 -- 172.21.15.146:0/3003321524 learned_addr learned my addr 172.21.15.146:0/3003321524 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T19:59:48.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:48.095+0000 7f02b4c3f700 1 -- 172.21.15.146:0/3003321524 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02a4005040 con 0x7f02b00f6510 2024-01-21T19:59:48.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:48.096+0000 7f02b4c3f700 1 --2- 172.21.15.146:0/3003321524 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02b00f6510 0x7f02b0106730 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f02a4013510 tx=0x7f02a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:48.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:48.096+0000 7f02ad7fa700 1 -- 172.21.15.146:0/3003321524 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02a4009320 con 0x7f02b00f6510 2024-01-21T19:59:48.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:48.096+0000 7f02b6ea3700 1 -- 172.21.15.146:0/3003321524 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02b0106e20 con 0x7f02b00f6510 2024-01-21T19:59:48.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:48.096+0000 7f02ad7fa700 1 -- 172.21.15.146:0/3003321524 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f02a4013f00 con 0x7f02b00f6510 2024-01-21T19:59:48.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:48.096+0000 7f02ad7fa700 1 -- 172.21.15.146:0/3003321524 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02a4014d90 con 0x7f02b00f6510 2024-01-21T19:59:48.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:48.096+0000 7f02b6ea3700 1 -- 172.21.15.146:0/3003321524 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02b01072b0 con 0x7f02b00f6510 2024-01-21T19:59:48.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:48.097+0000 7f02ad7fa700 1 -- 172.21.15.146:0/3003321524 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f02a40095f0 con 0x7f02b00f6510 2024-01-21T19:59:48.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:48.097+0000 7f02b6ea3700 1 -- 172.21.15.146:0/3003321524 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02b0045ad0 con 0x7f02b00f6510 2024-01-21T19:59:48.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:48.097+0000 7f02ad7fa700 1 --2- 172.21.15.146:0/3003321524 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f02a002ff60 0x7f02a0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:48.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:48.097+0000 7f02ad7fa700 1 -- 172.21.15.146:0/3003321524 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f02a4041080 con 0x7f02b00f6510 2024-01-21T19:59:48.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:48.097+0000 7f02affff700 1 --2- 172.21.15.146:0/3003321524 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f02a002ff60 0x7f02a0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:48.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:48.098+0000 7f02affff700 1 --2- 172.21.15.146:0/3003321524 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f02a002ff60 0x7f02a0032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f029c0096a0 tx=0x7f029c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:48.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:48.102+0000 7f02ad7fa700 1 -- 172.21.15.146:0/3003321524 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f02a4014420 con 0x7f02b00f6510 2024-01-21T19:59:48.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:48.308+0000 7f02b6ea3700 1 -- 172.21.15.146:0/3003321524 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f02b00fbf60 con 0x7f02b00f6510 2024-01-21T19:59:48.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:48.309+0000 7f02ad7fa700 1 -- 172.21.15.146:0/3003321524 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f02a4010230 con 0x7f02b00f6510 2024-01-21T19:59:48.300 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:59:48.300 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:59:48.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:48.310+0000 7f02b6ea3700 1 -- 172.21.15.146:0/3003321524 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f02a002ff60 msgr2=0x7f02a0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:48.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:48.310+0000 7f02b6ea3700 1 --2- 172.21.15.146:0/3003321524 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f02a002ff60 0x7f02a0032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f029c0096a0 tx=0x7f029c006b40).stop 2024-01-21T19:59:48.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:48.310+0000 7f02b6ea3700 1 -- 172.21.15.146:0/3003321524 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02b00f6510 msgr2=0x7f02b0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:48.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:48.310+0000 7f02b6ea3700 1 --2- 172.21.15.146:0/3003321524 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02b00f6510 0x7f02b0106730 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f02a4013510 tx=0x7f02a400b040).stop 2024-01-21T19:59:48.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:48.310+0000 7f02b6ea3700 1 -- 172.21.15.146:0/3003321524 shutdown_connections 2024-01-21T19:59:48.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:48.310+0000 7f02b6ea3700 1 --2- 172.21.15.146:0/3003321524 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f02a002ff60 0x7f02a0032420 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:48.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:48.310+0000 7f02b6ea3700 1 --2- 172.21.15.146:0/3003321524 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02b00f6510 0x7f02b0106730 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:48.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:48.310+0000 7f02b6ea3700 1 -- 172.21.15.146:0/3003321524 >> 172.21.15.146:0/3003321524 conn(0x7f02b00f1c70 msgr2=0x7f02b00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:48.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:48.310+0000 7f02b6ea3700 1 -- 172.21.15.146:0/3003321524 shutdown_connections 2024-01-21T19:59:48.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:48.310+0000 7f02b6ea3700 1 -- 172.21.15.146:0/3003321524 wait complete. 2024-01-21T19:59:48.303 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T19:59:48.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:48 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:59:47.128951+0000 mgr.smithi053.fgsmol (mgr.14164) 20 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:59:48.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:48 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:59:48.310120+0000 mon.smithi053 (mon.0) 123 : audit [DBG] from='client.? 172.21.15.146:0/3003321524' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:59:49.699 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:59:49.699 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:59:50.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:50 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:59:49.129205+0000 mgr.smithi053.fgsmol (mgr.14164) 21 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:59:50.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:50.991+0000 7fb4c20fc700 1 -- 172.21.15.146:0/3571111665 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4bc060200 msgr2=0x7fb4bc0f9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:50.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:50.991+0000 7fb4c20fc700 1 --2- 172.21.15.146:0/3571111665 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4bc060200 0x7fb4bc0f9170 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fb4ac009fb0 tx=0x7fb4ac009b30).stop 2024-01-21T19:59:50.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:50.991+0000 7fb4c20fc700 1 -- 172.21.15.146:0/3571111665 shutdown_connections 2024-01-21T19:59:50.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:50.991+0000 7fb4c20fc700 1 --2- 172.21.15.146:0/3571111665 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4bc060200 0x7fb4bc0f9170 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:50.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:50.991+0000 7fb4c20fc700 1 -- 172.21.15.146:0/3571111665 >> 172.21.15.146:0/3571111665 conn(0x7fb4bc0f0b70 msgr2=0x7fb4bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:50.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:50.991+0000 7fb4c20fc700 1 -- 172.21.15.146:0/3571111665 shutdown_connections 2024-01-21T19:59:50.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:50.991+0000 7fb4c20fc700 1 -- 172.21.15.146:0/3571111665 wait complete. 2024-01-21T19:59:50.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:50.992+0000 7fb4c20fc700 1 Processor -- start 2024-01-21T19:59:50.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:50.992+0000 7fb4c20fc700 1 -- start start 2024-01-21T19:59:50.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:50.992+0000 7fb4c20fc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4bc060200 0x7fb4bc1066a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:50.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:50.992+0000 7fb4c20fc700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4bc106b90 con 0x7fb4bc060200 2024-01-21T19:59:50.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:50.992+0000 7fb4bb7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4bc060200 0x7fb4bc1066a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:50.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:50.992+0000 7fb4bb7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4bc060200 0x7fb4bc1066a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:46484/0 (socket says 172.21.15.146:46484) 2024-01-21T19:59:50.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:50.992+0000 7fb4bb7fe700 1 -- 172.21.15.146:0/294380090 learned_addr learned my addr 172.21.15.146:0/294380090 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T19:59:50.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:50.993+0000 7fb4bb7fe700 1 -- 172.21.15.146:0/294380090 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4ac005040 con 0x7fb4bc060200 2024-01-21T19:59:50.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:50.993+0000 7fb4bb7fe700 1 --2- 172.21.15.146:0/294380090 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4bc060200 0x7fb4bc1066a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fb4ac013510 tx=0x7fb4ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:50.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:50.993+0000 7fb4b8ff9700 1 -- 172.21.15.146:0/294380090 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4ac009320 con 0x7fb4bc060200 2024-01-21T19:59:50.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:50.993+0000 7fb4c20fc700 1 -- 172.21.15.146:0/294380090 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4bc106d90 con 0x7fb4bc060200 2024-01-21T19:59:50.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:50.993+0000 7fb4b8ff9700 1 -- 172.21.15.146:0/294380090 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb4ac013f00 con 0x7fb4bc060200 2024-01-21T19:59:50.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:50.993+0000 7fb4b8ff9700 1 -- 172.21.15.146:0/294380090 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4ac014dc0 con 0x7fb4bc060200 2024-01-21T19:59:50.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:50.993+0000 7fb4c20fc700 1 -- 172.21.15.146:0/294380090 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4bc107220 con 0x7fb4bc060200 2024-01-21T19:59:50.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:50.994+0000 7fb4b8ff9700 1 -- 172.21.15.146:0/294380090 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb4ac01d3d0 con 0x7fb4bc060200 2024-01-21T19:59:50.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:50.994+0000 7fb4b8ff9700 1 --2- 172.21.15.146:0/294380090 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb4a802ff60 0x7fb4a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:50.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:50.994+0000 7fb4b8ff9700 1 -- 172.21.15.146:0/294380090 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb4ac041110 con 0x7fb4bc060200 2024-01-21T19:59:50.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:50.994+0000 7fb4c20fc700 1 -- 172.21.15.146:0/294380090 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4bc045ad0 con 0x7fb4bc060200 2024-01-21T19:59:50.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:50.994+0000 7fb4b2ffd700 1 --2- 172.21.15.146:0/294380090 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb4a802ff60 0x7fb4a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:50.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:50.994+0000 7fb4b2ffd700 1 --2- 172.21.15.146:0/294380090 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb4a802ff60 0x7fb4a8032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fb4a40096a0 tx=0x7fb4a4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:50.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:50.999+0000 7fb4b8ff9700 1 -- 172.21.15.146:0/294380090 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb4ac012920 con 0x7fb4bc060200 2024-01-21T19:59:51.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:51.203+0000 7fb4c20fc700 1 -- 172.21.15.146:0/294380090 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb4bc107f40 con 0x7fb4bc060200 2024-01-21T19:59:51.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:51.203+0000 7fb4b8ff9700 1 -- 172.21.15.146:0/294380090 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb4ac02b020 con 0x7fb4bc060200 2024-01-21T19:59:51.196 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:59:51.196 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:59:51.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:51.205+0000 7fb4c20fc700 1 -- 172.21.15.146:0/294380090 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb4a802ff60 msgr2=0x7fb4a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:51.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:51.205+0000 7fb4c20fc700 1 --2- 172.21.15.146:0/294380090 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb4a802ff60 0x7fb4a8032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fb4a40096a0 tx=0x7fb4a4006b40).stop 2024-01-21T19:59:51.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:51.205+0000 7fb4c20fc700 1 -- 172.21.15.146:0/294380090 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4bc060200 msgr2=0x7fb4bc1066a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:51.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:51.205+0000 7fb4c20fc700 1 --2- 172.21.15.146:0/294380090 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4bc060200 0x7fb4bc1066a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fb4ac013510 tx=0x7fb4ac00b040).stop 2024-01-21T19:59:51.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:51.205+0000 7fb4c20fc700 1 -- 172.21.15.146:0/294380090 shutdown_connections 2024-01-21T19:59:51.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:51.205+0000 7fb4c20fc700 1 --2- 172.21.15.146:0/294380090 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb4a802ff60 0x7fb4a8032420 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:51.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:51.205+0000 7fb4c20fc700 1 --2- 172.21.15.146:0/294380090 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4bc060200 0x7fb4bc1066a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:51.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:51.205+0000 7fb4c20fc700 1 -- 172.21.15.146:0/294380090 >> 172.21.15.146:0/294380090 conn(0x7fb4bc0f0b70 msgr2=0x7fb4bc0f1650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:51.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:51.205+0000 7fb4c20fc700 1 -- 172.21.15.146:0/294380090 shutdown_connections 2024-01-21T19:59:51.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:51.205+0000 7fb4c20fc700 1 -- 172.21.15.146:0/294380090 wait complete. 2024-01-21T19:59:51.198 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T19:59:51.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:51 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:59:51.204660+0000 mon.smithi053 (mon.0) 124 : audit [DBG] from='client.? 172.21.15.146:0/294380090' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:59:52.592 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:59:52.592 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:59:52.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:52 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:59:51.129476+0000 mgr.smithi053.fgsmol (mgr.14164) 22 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:59:53.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:53.876+0000 7f5f28356700 1 -- 172.21.15.146:0/768225749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f200f5410 msgr2=0x7f5f200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:53.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:53.876+0000 7f5f28356700 1 --2- 172.21.15.146:0/768225749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f200f5410 0x7f5f200f5830 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f5f1c009fb0 tx=0x7f5f1c009b30).stop 2024-01-21T19:59:53.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:53.876+0000 7f5f28356700 1 -- 172.21.15.146:0/768225749 shutdown_connections 2024-01-21T19:59:53.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:53.876+0000 7f5f28356700 1 --2- 172.21.15.146:0/768225749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f200f5410 0x7f5f200f5830 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:53.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:53.876+0000 7f5f28356700 1 -- 172.21.15.146:0/768225749 >> 172.21.15.146:0/768225749 conn(0x7f5f200f0b70 msgr2=0x7f5f200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:53.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:53.876+0000 7f5f28356700 1 -- 172.21.15.146:0/768225749 shutdown_connections 2024-01-21T19:59:53.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:53.876+0000 7f5f28356700 1 -- 172.21.15.146:0/768225749 wait complete. 2024-01-21T19:59:53.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:53.876+0000 7f5f28356700 1 Processor -- start 2024-01-21T19:59:53.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:53.876+0000 7f5f28356700 1 -- start start 2024-01-21T19:59:53.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:53.877+0000 7f5f28356700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f200f5410 0x7f5f20109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:53.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:53.877+0000 7f5f28356700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f20060240 con 0x7f5f200f5410 2024-01-21T19:59:53.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:53.877+0000 7f5f260f2700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f200f5410 0x7f5f20109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:53.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:53.877+0000 7f5f260f2700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f200f5410 0x7f5f20109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:46496/0 (socket says 172.21.15.146:46496) 2024-01-21T19:59:53.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:53.877+0000 7f5f260f2700 1 -- 172.21.15.146:0/601310844 learned_addr learned my addr 172.21.15.146:0/601310844 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T19:59:53.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:53.877+0000 7f5f260f2700 1 -- 172.21.15.146:0/601310844 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f1c005040 con 0x7f5f200f5410 2024-01-21T19:59:53.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:53.877+0000 7f5f260f2700 1 --2- 172.21.15.146:0/601310844 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f200f5410 0x7f5f20109430 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f5f1c014000 tx=0x7f5f1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:53.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:53.877+0000 7f5f16ffd700 1 -- 172.21.15.146:0/601310844 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f1c0095f0 con 0x7f5f200f5410 2024-01-21T19:59:53.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:53.877+0000 7f5f28356700 1 -- 172.21.15.146:0/601310844 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f20060440 con 0x7f5f200f5410 2024-01-21T19:59:53.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:53.877+0000 7f5f16ffd700 1 -- 172.21.15.146:0/601310844 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5f1c007e70 con 0x7f5f200f5410 2024-01-21T19:59:53.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:53.877+0000 7f5f16ffd700 1 -- 172.21.15.146:0/601310844 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f1c014ea0 con 0x7f5f200f5410 2024-01-21T19:59:53.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:53.877+0000 7f5f28356700 1 -- 172.21.15.146:0/601310844 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f20109ae0 con 0x7f5f200f5410 2024-01-21T19:59:53.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:53.878+0000 7f5f28356700 1 -- 172.21.15.146:0/601310844 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f20045ad0 con 0x7f5f200f5410 2024-01-21T19:59:53.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:53.878+0000 7f5f16ffd700 1 -- 172.21.15.146:0/601310844 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5f1c014690 con 0x7f5f200f5410 2024-01-21T19:59:53.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:53.878+0000 7f5f16ffd700 1 --2- 172.21.15.146:0/601310844 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f5f0c02ff60 0x7f5f0c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:53.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:53.878+0000 7f5f16ffd700 1 -- 172.21.15.146:0/601310844 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5f1c040930 con 0x7f5f200f5410 2024-01-21T19:59:53.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:53.879+0000 7f5f258f1700 1 --2- 172.21.15.146:0/601310844 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f5f0c02ff60 0x7f5f0c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:53.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:53.879+0000 7f5f258f1700 1 --2- 172.21.15.146:0/601310844 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f5f0c02ff60 0x7f5f0c032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f5f100096a0 tx=0x7f5f10006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:53.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:53.883+0000 7f5f16ffd700 1 -- 172.21.15.146:0/601310844 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5f1c00f300 con 0x7f5f200f5410 2024-01-21T19:59:54.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:54.086+0000 7f5f28356700 1 -- 172.21.15.146:0/601310844 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5f200fae60 con 0x7f5f200f5410 2024-01-21T19:59:54.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:54.087+0000 7f5f16ffd700 1 -- 172.21.15.146:0/601310844 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5f1c00f300 con 0x7f5f200f5410 2024-01-21T19:59:54.079 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:59:54.079 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:59:54.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:54.089+0000 7f5f28356700 1 -- 172.21.15.146:0/601310844 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f5f0c02ff60 msgr2=0x7f5f0c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:54.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:54.089+0000 7f5f28356700 1 --2- 172.21.15.146:0/601310844 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f5f0c02ff60 0x7f5f0c032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f5f100096a0 tx=0x7f5f10006b40).stop 2024-01-21T19:59:54.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:54.089+0000 7f5f28356700 1 -- 172.21.15.146:0/601310844 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f200f5410 msgr2=0x7f5f20109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:54.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:54.089+0000 7f5f28356700 1 --2- 172.21.15.146:0/601310844 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f200f5410 0x7f5f20109430 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f5f1c014000 tx=0x7f5f1c00b040).stop 2024-01-21T19:59:54.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:54.089+0000 7f5f28356700 1 -- 172.21.15.146:0/601310844 shutdown_connections 2024-01-21T19:59:54.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:54.089+0000 7f5f28356700 1 --2- 172.21.15.146:0/601310844 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f5f0c02ff60 0x7f5f0c032420 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:54.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:54.089+0000 7f5f28356700 1 --2- 172.21.15.146:0/601310844 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f200f5410 0x7f5f20109430 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:54.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:54.089+0000 7f5f28356700 1 -- 172.21.15.146:0/601310844 >> 172.21.15.146:0/601310844 conn(0x7f5f200f0b70 msgr2=0x7f5f200f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:54.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:54.089+0000 7f5f28356700 1 -- 172.21.15.146:0/601310844 shutdown_connections 2024-01-21T19:59:54.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:54.089+0000 7f5f28356700 1 -- 172.21.15.146:0/601310844 wait complete. 2024-01-21T19:59:54.081 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T19:59:54.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:54 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:59:53.129820+0000 mgr.smithi053.fgsmol (mgr.14164) 23 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:59:54.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:54 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:59:54.088490+0000 mon.smithi053 (mon.0) 125 : audit [DBG] from='client.? 172.21.15.146:0/601310844' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:59:55.472 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:59:55.473 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:59:56.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:56.750+0000 7f13611cc700 1 -- 172.21.15.146:0/3917094134 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f135c0f5410 msgr2=0x7f135c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:56.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:56.750+0000 7f13611cc700 1 --2- 172.21.15.146:0/3917094134 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f135c0f5410 0x7f135c0f5830 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f1344009fb0 tx=0x7f1344009b30).stop 2024-01-21T19:59:56.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:56.750+0000 7f13611cc700 1 -- 172.21.15.146:0/3917094134 shutdown_connections 2024-01-21T19:59:56.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:56.750+0000 7f13611cc700 1 --2- 172.21.15.146:0/3917094134 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f135c0f5410 0x7f135c0f5830 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:56.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:56.750+0000 7f13611cc700 1 -- 172.21.15.146:0/3917094134 >> 172.21.15.146:0/3917094134 conn(0x7f135c0f0b70 msgr2=0x7f135c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:56.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:56.750+0000 7f13611cc700 1 -- 172.21.15.146:0/3917094134 shutdown_connections 2024-01-21T19:59:56.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:56.750+0000 7f13611cc700 1 -- 172.21.15.146:0/3917094134 wait complete. 2024-01-21T19:59:56.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:56.751+0000 7f13611cc700 1 Processor -- start 2024-01-21T19:59:56.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:56.751+0000 7f13611cc700 1 -- start start 2024-01-21T19:59:56.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:56.751+0000 7f13611cc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f135c0f5410 0x7f135c1093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:56.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:56.751+0000 7f13611cc700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f135c060280 con 0x7f135c0f5410 2024-01-21T19:59:56.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:56.751+0000 7f135ad9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f135c0f5410 0x7f135c1093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:56.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:56.751+0000 7f135ad9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f135c0f5410 0x7f135c1093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:46510/0 (socket says 172.21.15.146:46510) 2024-01-21T19:59:56.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:56.751+0000 7f135ad9d700 1 -- 172.21.15.146:0/1456281711 learned_addr learned my addr 172.21.15.146:0/1456281711 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T19:59:56.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:56.752+0000 7f135ad9d700 1 -- 172.21.15.146:0/1456281711 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1344005040 con 0x7f135c0f5410 2024-01-21T19:59:56.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:56.752+0000 7f135ad9d700 1 --2- 172.21.15.146:0/1456281711 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f135c0f5410 0x7f135c1093f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f1344014000 tx=0x7f134400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:56.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:56.752+0000 7f13537fe700 1 -- 172.21.15.146:0/1456281711 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13440095f0 con 0x7f135c0f5410 2024-01-21T19:59:56.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:56.752+0000 7f13611cc700 1 -- 172.21.15.146:0/1456281711 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f135c060480 con 0x7f135c0f5410 2024-01-21T19:59:56.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:56.752+0000 7f13537fe700 1 -- 172.21.15.146:0/1456281711 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1344007e70 con 0x7f135c0f5410 2024-01-21T19:59:56.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:56.752+0000 7f13537fe700 1 -- 172.21.15.146:0/1456281711 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1344014ea0 con 0x7f135c0f5410 2024-01-21T19:59:56.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:56.752+0000 7f13611cc700 1 -- 172.21.15.146:0/1456281711 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f135c109b20 con 0x7f135c0f5410 2024-01-21T19:59:56.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:56.753+0000 7f13537fe700 1 -- 172.21.15.146:0/1456281711 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1344025070 con 0x7f135c0f5410 2024-01-21T19:59:56.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:56.753+0000 7f13611cc700 1 -- 172.21.15.146:0/1456281711 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f135c045ad0 con 0x7f135c0f5410 2024-01-21T19:59:56.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:56.753+0000 7f13537fe700 1 --2- 172.21.15.146:0/1456281711 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f134802ff60 0x7f1348032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:56.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:56.753+0000 7f13537fe700 1 -- 172.21.15.146:0/1456281711 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f13440409f0 con 0x7f135c0f5410 2024-01-21T19:59:56.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:56.753+0000 7f135a59c700 1 --2- 172.21.15.146:0/1456281711 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f134802ff60 0x7f1348032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:56.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:56.754+0000 7f135a59c700 1 --2- 172.21.15.146:0/1456281711 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f134802ff60 0x7f1348032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f134c0096a0 tx=0x7f134c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:56.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:56.758+0000 7f13537fe700 1 -- 172.21.15.146:0/1456281711 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f134400f4e0 con 0x7f135c0f5410 2024-01-21T19:59:56.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:56 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:59:55.130176+0000 mgr.smithi053.fgsmol (mgr.14164) 24 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:59:56.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:56.961+0000 7f13611cc700 1 -- 172.21.15.146:0/1456281711 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f135c10a340 con 0x7f135c0f5410 2024-01-21T19:59:56.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:56.962+0000 7f13537fe700 1 -- 172.21.15.146:0/1456281711 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f134400f300 con 0x7f135c0f5410 2024-01-21T19:59:56.957 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:59:56.957 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:59:56.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:56.964+0000 7f13611cc700 1 -- 172.21.15.146:0/1456281711 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f134802ff60 msgr2=0x7f1348032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:56.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:56.964+0000 7f13611cc700 1 --2- 172.21.15.146:0/1456281711 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f134802ff60 0x7f1348032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f134c0096a0 tx=0x7f134c006b40).stop 2024-01-21T19:59:56.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:56.964+0000 7f13611cc700 1 -- 172.21.15.146:0/1456281711 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f135c0f5410 msgr2=0x7f135c1093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:56.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:56.964+0000 7f13611cc700 1 --2- 172.21.15.146:0/1456281711 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f135c0f5410 0x7f135c1093f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f1344014000 tx=0x7f134400b040).stop 2024-01-21T19:59:56.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:56.964+0000 7f13611cc700 1 -- 172.21.15.146:0/1456281711 shutdown_connections 2024-01-21T19:59:56.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:56.964+0000 7f13611cc700 1 --2- 172.21.15.146:0/1456281711 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f134802ff60 0x7f1348032420 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:56.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:56.964+0000 7f13611cc700 1 --2- 172.21.15.146:0/1456281711 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f135c0f5410 0x7f135c1093f0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:56.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:56.964+0000 7f13611cc700 1 -- 172.21.15.146:0/1456281711 >> 172.21.15.146:0/1456281711 conn(0x7f135c0f0b70 msgr2=0x7f135c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:56.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:56.964+0000 7f13611cc700 1 -- 172.21.15.146:0/1456281711 shutdown_connections 2024-01-21T19:59:56.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:56.964+0000 7f13611cc700 1 -- 172.21.15.146:0/1456281711 wait complete. 2024-01-21T19:59:56.960 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T19:59:57.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:57 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T19:59:56.963481+0000 mon.smithi053 (mon.0) 126 : audit [DBG] from='client.? 172.21.15.146:0/1456281711' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:59:58.375 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:59:58.376 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:59:58.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 19:59:58 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:59:57.130558+0000 mgr.smithi053.fgsmol (mgr.14164) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:59:59.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:59.789+0000 7fc085ba0700 1 -- 172.21.15.146:0/2257905054 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0800f62a0 msgr2=0x7fc0800f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:59.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:59.789+0000 7fc085ba0700 1 --2- 172.21.15.146:0/2257905054 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0800f62a0 0x7fc0800f66c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fc074009fb0 tx=0x7fc074009b30).stop 2024-01-21T19:59:59.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:59.789+0000 7fc085ba0700 1 -- 172.21.15.146:0/2257905054 shutdown_connections 2024-01-21T19:59:59.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:59.789+0000 7fc085ba0700 1 --2- 172.21.15.146:0/2257905054 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0800f62a0 0x7fc0800f66c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:59.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:59.789+0000 7fc085ba0700 1 -- 172.21.15.146:0/2257905054 >> 172.21.15.146:0/2257905054 conn(0x7fc0800f1aa0 msgr2=0x7fc0800f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:59.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:59.789+0000 7fc085ba0700 1 -- 172.21.15.146:0/2257905054 shutdown_connections 2024-01-21T19:59:59.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:59.789+0000 7fc085ba0700 1 -- 172.21.15.146:0/2257905054 wait complete. 2024-01-21T19:59:59.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:59.790+0000 7fc085ba0700 1 Processor -- start 2024-01-21T19:59:59.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:59.790+0000 7fc085ba0700 1 -- start start 2024-01-21T19:59:59.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:59.790+0000 7fc085ba0700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0800f62a0 0x7fc0801064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:59.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:59.790+0000 7fc085ba0700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0801069a0 con 0x7fc0800f62a0 2024-01-21T19:59:59.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:59.790+0000 7fc084b9e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0800f62a0 0x7fc0801064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:59.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:59.790+0000 7fc084b9e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0800f62a0 0x7fc0801064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:50024/0 (socket says 172.21.15.146:50024) 2024-01-21T19:59:59.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:59.790+0000 7fc084b9e700 1 -- 172.21.15.146:0/1029469452 learned_addr learned my addr 172.21.15.146:0/1029469452 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T19:59:59.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:59.791+0000 7fc084b9e700 1 -- 172.21.15.146:0/1029469452 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc074005040 con 0x7fc0800f62a0 2024-01-21T19:59:59.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:59.791+0000 7fc084b9e700 1 --2- 172.21.15.146:0/1029469452 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0800f62a0 0x7fc0801064b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fc074013510 tx=0x7fc07400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:59.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:59.791+0000 7fc07d7fa700 1 -- 172.21.15.146:0/1029469452 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc074009320 con 0x7fc0800f62a0 2024-01-21T19:59:59.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:59.791+0000 7fc085ba0700 1 -- 172.21.15.146:0/1029469452 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc080106ba0 con 0x7fc0800f62a0 2024-01-21T19:59:59.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:59.791+0000 7fc07d7fa700 1 -- 172.21.15.146:0/1029469452 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc074013f00 con 0x7fc0800f62a0 2024-01-21T19:59:59.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:59.791+0000 7fc07d7fa700 1 -- 172.21.15.146:0/1029469452 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc074014dc0 con 0x7fc0800f62a0 2024-01-21T19:59:59.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:59.791+0000 7fc085ba0700 1 -- 172.21.15.146:0/1029469452 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc080107030 con 0x7fc0800f62a0 2024-01-21T19:59:59.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:59.792+0000 7fc07d7fa700 1 -- 172.21.15.146:0/1029469452 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc074014420 con 0x7fc0800f62a0 2024-01-21T19:59:59.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:59.792+0000 7fc085ba0700 1 -- 172.21.15.146:0/1029469452 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc080045ad0 con 0x7fc0800f62a0 2024-01-21T19:59:59.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:59.792+0000 7fc07d7fa700 1 --2- 172.21.15.146:0/1029469452 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fc07002ff60 0x7fc070032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:59.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:59.792+0000 7fc07d7fa700 1 -- 172.21.15.146:0/1029469452 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc074041030 con 0x7fc0800f62a0 2024-01-21T19:59:59.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:59.792+0000 7fc07ffff700 1 --2- 172.21.15.146:0/1029469452 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fc07002ff60 0x7fc070032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:59.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:59.793+0000 7fc07ffff700 1 --2- 172.21.15.146:0/1029469452 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fc07002ff60 0x7fc070032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fc06c0096a0 tx=0x7fc06c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:59.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T19:59:59.797+0000 7fc07d7fa700 1 -- 172.21.15.146:0/1029469452 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc0740095f0 con 0x7fc0800f62a0 2024-01-21T19:59:59.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:00.001+0000 7fc085ba0700 1 -- 172.21.15.146:0/1029469452 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc080107ee0 con 0x7fc0800f62a0 2024-01-21T19:59:59.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:00.002+0000 7fc07d7fa700 1 -- 172.21.15.146:0/1029469452 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc07401cb10 con 0x7fc0800f62a0 2024-01-21T19:59:59.994 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T19:59:59.994 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:59:59.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:00.004+0000 7fc085ba0700 1 -- 172.21.15.146:0/1029469452 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fc07002ff60 msgr2=0x7fc070032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:59.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:00.004+0000 7fc085ba0700 1 --2- 172.21.15.146:0/1029469452 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fc07002ff60 0x7fc070032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fc06c0096a0 tx=0x7fc06c006b40).stop 2024-01-21T19:59:59.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:00.004+0000 7fc085ba0700 1 -- 172.21.15.146:0/1029469452 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0800f62a0 msgr2=0x7fc0801064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:59.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:00.004+0000 7fc085ba0700 1 --2- 172.21.15.146:0/1029469452 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0800f62a0 0x7fc0801064b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fc074013510 tx=0x7fc07400b040).stop 2024-01-21T19:59:59.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:00.004+0000 7fc085ba0700 1 -- 172.21.15.146:0/1029469452 shutdown_connections 2024-01-21T19:59:59.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:00.004+0000 7fc085ba0700 1 --2- 172.21.15.146:0/1029469452 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fc07002ff60 0x7fc070032420 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:59.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:00.004+0000 7fc085ba0700 1 --2- 172.21.15.146:0/1029469452 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0800f62a0 0x7fc0801064b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:59.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:00.004+0000 7fc085ba0700 1 -- 172.21.15.146:0/1029469452 >> 172.21.15.146:0/1029469452 conn(0x7fc0800f1aa0 msgr2=0x7fc0800f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:59.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:00.004+0000 7fc085ba0700 1 -- 172.21.15.146:0/1029469452 shutdown_connections 2024-01-21T19:59:59.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:00.004+0000 7fc085ba0700 1 -- 172.21.15.146:0/1029469452 wait complete. 2024-01-21T19:59:59.996 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:00:00.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:00 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T19:59:59.130943+0000 mgr.smithi053.fgsmol (mgr.14164) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:00.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:00 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:00:00.003469+0000 mon.smithi053 (mon.0) 127 : audit [DBG] from='client.? 172.21.15.146:0/1029469452' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:00:01.404 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:00:01.405 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:00:02.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:02.580+0000 7fe7f8f9a700 1 -- 172.21.15.146:0/1193769206 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7f40f6510 msgr2=0x7fe7f40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:02.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:02.580+0000 7fe7f8f9a700 1 --2- 172.21.15.146:0/1193769206 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7f40f6510 0x7fe7f40f6930 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fe7dc009fb0 tx=0x7fe7dc009b30).stop 2024-01-21T20:00:02.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:02.580+0000 7fe7f8f9a700 1 -- 172.21.15.146:0/1193769206 shutdown_connections 2024-01-21T20:00:02.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:02.580+0000 7fe7f8f9a700 1 --2- 172.21.15.146:0/1193769206 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7f40f6510 0x7fe7f40f6930 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:02.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:02.580+0000 7fe7f8f9a700 1 -- 172.21.15.146:0/1193769206 >> 172.21.15.146:0/1193769206 conn(0x7fe7f40f1c70 msgr2=0x7fe7f40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:02.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:02.580+0000 7fe7f8f9a700 1 -- 172.21.15.146:0/1193769206 shutdown_connections 2024-01-21T20:00:02.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:02.580+0000 7fe7f8f9a700 1 -- 172.21.15.146:0/1193769206 wait complete. 2024-01-21T20:00:02.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:02.581+0000 7fe7f8f9a700 1 Processor -- start 2024-01-21T20:00:02.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:02.581+0000 7fe7f8f9a700 1 -- start start 2024-01-21T20:00:02.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:02.581+0000 7fe7f8f9a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7f40f6510 0x7fe7f406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:02.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:02.581+0000 7fe7f8f9a700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f4069e20 con 0x7fe7f40f6510 2024-01-21T20:00:02.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:02.581+0000 7fe7f259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7f40f6510 0x7fe7f406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:02.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:02.581+0000 7fe7f259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7f40f6510 0x7fe7f406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:50040/0 (socket says 172.21.15.146:50040) 2024-01-21T20:00:02.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:02.581+0000 7fe7f259c700 1 -- 172.21.15.146:0/32058389 learned_addr learned my addr 172.21.15.146:0/32058389 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:00:02.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:02.581+0000 7fe7f259c700 1 -- 172.21.15.146:0/32058389 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7dc005040 con 0x7fe7f40f6510 2024-01-21T20:00:02.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:02.581+0000 7fe7f259c700 1 --2- 172.21.15.146:0/32058389 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7f40f6510 0x7fe7f406b9a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fe7dc014000 tx=0x7fe7dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:02.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:02.582+0000 7fe7eaffd700 1 -- 172.21.15.146:0/32058389 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7dc0095f0 con 0x7fe7f40f6510 2024-01-21T20:00:02.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:02.582+0000 7fe7f8f9a700 1 -- 172.21.15.146:0/32058389 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7f4069fc0 con 0x7fe7f40f6510 2024-01-21T20:00:02.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:02.582+0000 7fe7eaffd700 1 -- 172.21.15.146:0/32058389 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe7dc007e70 con 0x7fe7f40f6510 2024-01-21T20:00:02.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:02.582+0000 7fe7eaffd700 1 -- 172.21.15.146:0/32058389 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7dc014ea0 con 0x7fe7f40f6510 2024-01-21T20:00:02.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:02.582+0000 7fe7f8f9a700 1 -- 172.21.15.146:0/32058389 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7f406a450 con 0x7fe7f40f6510 2024-01-21T20:00:02.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:02.582+0000 7fe7eaffd700 1 -- 172.21.15.146:0/32058389 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe7dc025070 con 0x7fe7f40f6510 2024-01-21T20:00:02.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:02.583+0000 7fe7eaffd700 1 --2- 172.21.15.146:0/32058389 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fe7e002ff60 0x7fe7e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:02.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:02.583+0000 7fe7eaffd700 1 -- 172.21.15.146:0/32058389 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe7dc040d40 con 0x7fe7f40f6510 2024-01-21T20:00:02.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:02.583+0000 7fe7f8f9a700 1 -- 172.21.15.146:0/32058389 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7f4059100 con 0x7fe7f40f6510 2024-01-21T20:00:02.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:02.583+0000 7fe7f1d9b700 1 --2- 172.21.15.146:0/32058389 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fe7e002ff60 0x7fe7e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:02.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:02.583+0000 7fe7f1d9b700 1 --2- 172.21.15.146:0/32058389 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fe7e002ff60 0x7fe7e0032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fe7e40096a0 tx=0x7fe7e4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:02.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:02.588+0000 7fe7eaffd700 1 -- 172.21.15.146:0/32058389 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe7dc026350 con 0x7fe7f40f6510 2024-01-21T20:00:02.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:02.796+0000 7fe7f8f9a700 1 -- 172.21.15.146:0/32058389 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe7f40fbf60 con 0x7fe7f40f6510 2024-01-21T20:00:02.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:02.796+0000 7fe7eaffd700 1 -- 172.21.15.146:0/32058389 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe7dc012330 con 0x7fe7f40f6510 2024-01-21T20:00:02.788 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:00:02.788 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:00:02.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:02.798+0000 7fe7f8f9a700 1 -- 172.21.15.146:0/32058389 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fe7e002ff60 msgr2=0x7fe7e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:02.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:02.798+0000 7fe7f8f9a700 1 --2- 172.21.15.146:0/32058389 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fe7e002ff60 0x7fe7e0032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fe7e40096a0 tx=0x7fe7e4006b40).stop 2024-01-21T20:00:02.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:02.798+0000 7fe7f8f9a700 1 -- 172.21.15.146:0/32058389 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7f40f6510 msgr2=0x7fe7f406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:02.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:02.798+0000 7fe7f8f9a700 1 --2- 172.21.15.146:0/32058389 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7f40f6510 0x7fe7f406b9a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fe7dc014000 tx=0x7fe7dc00b040).stop 2024-01-21T20:00:02.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:02.798+0000 7fe7f8f9a700 1 -- 172.21.15.146:0/32058389 shutdown_connections 2024-01-21T20:00:02.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:02.798+0000 7fe7f8f9a700 1 --2- 172.21.15.146:0/32058389 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fe7e002ff60 0x7fe7e0032420 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:02.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:02.798+0000 7fe7f8f9a700 1 --2- 172.21.15.146:0/32058389 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7f40f6510 0x7fe7f406b9a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:02.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:02.798+0000 7fe7f8f9a700 1 -- 172.21.15.146:0/32058389 >> 172.21.15.146:0/32058389 conn(0x7fe7f40f1c70 msgr2=0x7fe7f40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:02.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:02.798+0000 7fe7f8f9a700 1 -- 172.21.15.146:0/32058389 shutdown_connections 2024-01-21T20:00:02.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:02.798+0000 7fe7f8f9a700 1 -- 172.21.15.146:0/32058389 wait complete. 2024-01-21T20:00:02.791 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:00:02.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:02 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:00:01.131250+0000 mgr.smithi053.fgsmol (mgr.14164) 27 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:03.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:03 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:00:02.797607+0000 mon.smithi053 (mon.0) 128 : audit [DBG] from='client.? 172.21.15.146:0/32058389' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:00:04.214 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:00:04.215 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:00:04.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:04 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:00:03.131589+0000 mgr.smithi053.fgsmol (mgr.14164) 28 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:06.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:06 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:00:05.131857+0000 mgr.smithi053.fgsmol (mgr.14164) 29 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:08.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:08 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:00:07.132208+0000 mgr.smithi053.fgsmol (mgr.14164) 30 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:09.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:09.600+0000 7f078c83c700 1 -- 172.21.15.146:0/1700300230 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07880f5410 msgr2=0x7f07880f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:09.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:09.600+0000 7f078c83c700 1 --2- 172.21.15.146:0/1700300230 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07880f5410 0x7f07880f5830 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f0778009fb0 tx=0x7f0778009b30).stop 2024-01-21T20:00:09.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:09.600+0000 7f078c83c700 1 -- 172.21.15.146:0/1700300230 shutdown_connections 2024-01-21T20:00:09.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:09.600+0000 7f078c83c700 1 --2- 172.21.15.146:0/1700300230 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07880f5410 0x7f07880f5830 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:09.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:09.600+0000 7f078c83c700 1 -- 172.21.15.146:0/1700300230 >> 172.21.15.146:0/1700300230 conn(0x7f07880f0b70 msgr2=0x7f07880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:09.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:09.600+0000 7f078c83c700 1 -- 172.21.15.146:0/1700300230 shutdown_connections 2024-01-21T20:00:09.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:09.600+0000 7f078c83c700 1 -- 172.21.15.146:0/1700300230 wait complete. 2024-01-21T20:00:09.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:09.600+0000 7f078c83c700 1 Processor -- start 2024-01-21T20:00:09.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:09.600+0000 7f078c83c700 1 -- start start 2024-01-21T20:00:09.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:09.600+0000 7f078c83c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07880f5410 0x7f078810b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:09.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:09.600+0000 7f078c83c700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07881071e0 con 0x7f07880f5410 2024-01-21T20:00:09.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:09.601+0000 7f078659c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07880f5410 0x7f078810b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:09.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:09.601+0000 7f078659c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07880f5410 0x7f078810b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:46832/0 (socket says 172.21.15.146:46832) 2024-01-21T20:00:09.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:09.601+0000 7f078659c700 1 -- 172.21.15.146:0/1157417647 learned_addr learned my addr 172.21.15.146:0/1157417647 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:00:09.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:09.601+0000 7f078659c700 1 -- 172.21.15.146:0/1157417647 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0778005040 con 0x7f07880f5410 2024-01-21T20:00:09.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:09.601+0000 7f078659c700 1 --2- 172.21.15.146:0/1157417647 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07880f5410 0x7f078810b7f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f0778013510 tx=0x7f077800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:09.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:09.601+0000 7f0776ffd700 1 -- 172.21.15.146:0/1157417647 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0778009320 con 0x7f07880f5410 2024-01-21T20:00:09.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:09.601+0000 7f0776ffd700 1 -- 172.21.15.146:0/1157417647 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0778013f00 con 0x7f07880f5410 2024-01-21T20:00:09.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:09.601+0000 7f0776ffd700 1 -- 172.21.15.146:0/1157417647 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0778014d90 con 0x7f07880f5410 2024-01-21T20:00:09.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:09.601+0000 7f078c83c700 1 -- 172.21.15.146:0/1157417647 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0788060280 con 0x7f07880f5410 2024-01-21T20:00:09.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:09.601+0000 7f078c83c700 1 -- 172.21.15.146:0/1157417647 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f078810bce0 con 0x7f07880f5410 2024-01-21T20:00:09.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:09.602+0000 7f0776ffd700 1 -- 172.21.15.146:0/1157417647 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f07780095f0 con 0x7f07880f5410 2024-01-21T20:00:09.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:09.602+0000 7f078c83c700 1 -- 172.21.15.146:0/1157417647 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0788045ad0 con 0x7f07880f5410 2024-01-21T20:00:09.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:09.602+0000 7f0776ffd700 1 --2- 172.21.15.146:0/1157417647 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f077002ff60 0x7f0770032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:09.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:09.602+0000 7f0776ffd700 1 -- 172.21.15.146:0/1157417647 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0778041080 con 0x7f07880f5410 2024-01-21T20:00:09.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:09.603+0000 7f0785d9b700 1 --2- 172.21.15.146:0/1157417647 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f077002ff60 0x7f0770032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:09.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:09.603+0000 7f0785d9b700 1 --2- 172.21.15.146:0/1157417647 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f077002ff60 0x7f0770032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f077c0096a0 tx=0x7f077c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:09.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:09.607+0000 7f0776ffd700 1 -- 172.21.15.146:0/1157417647 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f077801d7f0 con 0x7f07880f5410 2024-01-21T20:00:09.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:09.813+0000 7f078c83c700 1 -- 172.21.15.146:0/1157417647 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f07880fae60 con 0x7f07880f5410 2024-01-21T20:00:09.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:09.814+0000 7f0776ffd700 1 -- 172.21.15.146:0/1157417647 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0778010230 con 0x7f07880f5410 2024-01-21T20:00:09.806 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:00:09.806 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:00:09.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:09.816+0000 7f078c83c700 1 -- 172.21.15.146:0/1157417647 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f077002ff60 msgr2=0x7f0770032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:09.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:09.816+0000 7f078c83c700 1 --2- 172.21.15.146:0/1157417647 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f077002ff60 0x7f0770032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f077c0096a0 tx=0x7f077c006b40).stop 2024-01-21T20:00:09.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:09.816+0000 7f078c83c700 1 -- 172.21.15.146:0/1157417647 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07880f5410 msgr2=0x7f078810b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:09.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:09.816+0000 7f078c83c700 1 --2- 172.21.15.146:0/1157417647 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07880f5410 0x7f078810b7f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f0778013510 tx=0x7f077800b040).stop 2024-01-21T20:00:09.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:09.816+0000 7f078c83c700 1 -- 172.21.15.146:0/1157417647 shutdown_connections 2024-01-21T20:00:09.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:09.816+0000 7f078c83c700 1 --2- 172.21.15.146:0/1157417647 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f077002ff60 0x7f0770032420 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:09.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:09.816+0000 7f078c83c700 1 --2- 172.21.15.146:0/1157417647 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07880f5410 0x7f078810b7f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:09.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:09.816+0000 7f078c83c700 1 -- 172.21.15.146:0/1157417647 >> 172.21.15.146:0/1157417647 conn(0x7f07880f0b70 msgr2=0x7f07880f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:09.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:09.816+0000 7f078c83c700 1 -- 172.21.15.146:0/1157417647 shutdown_connections 2024-01-21T20:00:09.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:09.816+0000 7f078c83c700 1 -- 172.21.15.146:0/1157417647 wait complete. 2024-01-21T20:00:09.808 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:00:09.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:09 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:00:09.262944+0000 mon.smithi053 (mon.0) 129 : audit [INF] from='mgr.14164 172.21.15.53:0/2331671680' entity='mgr.smithi053.fgsmol' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fgsmol/mirror_snapshot_schedule"}]: dispatch 2024-01-21T20:00:09.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:09 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:00:09.271513+0000 mon.smithi053 (mon.0) 130 : audit [INF] from='mgr.14164 172.21.15.53:0/2331671680' entity='mgr.smithi053.fgsmol' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fgsmol/trash_purge_schedule"}]: dispatch 2024-01-21T20:00:10.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:10 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:00:09.132525+0000 mgr.smithi053.fgsmol (mgr.14164) 31 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:10.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:10 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:00:09.815362+0000 mon.smithi053 (mon.0) 131 : audit [DBG] from='client.? 172.21.15.146:0/1157417647' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:00:12.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:12 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:00:11.132834+0000 mgr.smithi053.fgsmol (mgr.14164) 32 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:13.206 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:00:13.206 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:00:14.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:14.438+0000 7f968730e700 1 -- 172.21.15.146:0/1731128009 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96800f6510 msgr2=0x7f96800f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:14.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:14.438+0000 7f968730e700 1 --2- 172.21.15.146:0/1731128009 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96800f6510 0x7f96800f6930 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f9670009fb0 tx=0x7f9670009b30).stop 2024-01-21T20:00:14.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:14.438+0000 7f968730e700 1 -- 172.21.15.146:0/1731128009 shutdown_connections 2024-01-21T20:00:14.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:14.438+0000 7f968730e700 1 --2- 172.21.15.146:0/1731128009 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96800f6510 0x7f96800f6930 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:14.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:14.438+0000 7f968730e700 1 -- 172.21.15.146:0/1731128009 >> 172.21.15.146:0/1731128009 conn(0x7f96800f1c70 msgr2=0x7f96800f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:14.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:14.438+0000 7f968730e700 1 -- 172.21.15.146:0/1731128009 shutdown_connections 2024-01-21T20:00:14.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:14.438+0000 7f968730e700 1 -- 172.21.15.146:0/1731128009 wait complete. 2024-01-21T20:00:14.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:14.438+0000 7f968730e700 1 Processor -- start 2024-01-21T20:00:14.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:14.438+0000 7f968730e700 1 -- start start 2024-01-21T20:00:14.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:14.438+0000 7f968730e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96800f6510 0x7f9680106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:14.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:14.438+0000 7f968730e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9680106c20 con 0x7f96800f6510 2024-01-21T20:00:14.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:14.439+0000 7f96850aa700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96800f6510 0x7f9680106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:14.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:14.439+0000 7f96850aa700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96800f6510 0x7f9680106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:46860/0 (socket says 172.21.15.146:46860) 2024-01-21T20:00:14.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:14.439+0000 7f96850aa700 1 -- 172.21.15.146:0/2530642396 learned_addr learned my addr 172.21.15.146:0/2530642396 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:00:14.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:14.439+0000 7f96850aa700 1 -- 172.21.15.146:0/2530642396 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9670005040 con 0x7f96800f6510 2024-01-21T20:00:14.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:14.439+0000 7f96850aa700 1 --2- 172.21.15.146:0/2530642396 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96800f6510 0x7f9680106730 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f9670013510 tx=0x7f967000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:14.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:14.439+0000 7f9675ffb700 1 -- 172.21.15.146:0/2530642396 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9670009320 con 0x7f96800f6510 2024-01-21T20:00:14.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:14.439+0000 7f968730e700 1 -- 172.21.15.146:0/2530642396 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9680106e20 con 0x7f96800f6510 2024-01-21T20:00:14.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:14.439+0000 7f9675ffb700 1 -- 172.21.15.146:0/2530642396 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9670013f00 con 0x7f96800f6510 2024-01-21T20:00:14.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:14.439+0000 7f9675ffb700 1 -- 172.21.15.146:0/2530642396 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9670014d90 con 0x7f96800f6510 2024-01-21T20:00:14.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:14.440+0000 7f968730e700 1 -- 172.21.15.146:0/2530642396 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96801072b0 con 0x7f96800f6510 2024-01-21T20:00:14.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:14.440+0000 7f9675ffb700 1 -- 172.21.15.146:0/2530642396 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f96700095f0 con 0x7f96800f6510 2024-01-21T20:00:14.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:14.440+0000 7f968730e700 1 -- 172.21.15.146:0/2530642396 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9680045ad0 con 0x7f96800f6510 2024-01-21T20:00:14.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:14.440+0000 7f9675ffb700 1 --2- 172.21.15.146:0/2530642396 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f966c02ff60 0x7f966c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:14.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:14.441+0000 7f9675ffb700 1 -- 172.21.15.146:0/2530642396 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f96700410c0 con 0x7f96800f6510 2024-01-21T20:00:14.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:14.441+0000 7f96848a9700 1 --2- 172.21.15.146:0/2530642396 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f966c02ff60 0x7f966c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:14.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:14.441+0000 7f96848a9700 1 --2- 172.21.15.146:0/2530642396 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f966c02ff60 0x7f966c032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f967c0096a0 tx=0x7f967c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:14.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:14.446+0000 7f9675ffb700 1 -- 172.21.15.146:0/2530642396 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9670014420 con 0x7f96800f6510 2024-01-21T20:00:14.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:14.648+0000 7f968730e700 1 -- 172.21.15.146:0/2530642396 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f96800fbf60 con 0x7f96800f6510 2024-01-21T20:00:14.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:14.649+0000 7f9675ffb700 1 -- 172.21.15.146:0/2530642396 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f967001d7f0 con 0x7f96800f6510 2024-01-21T20:00:14.640 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:00:14.641 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:00:14.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:14.650+0000 7f968730e700 1 -- 172.21.15.146:0/2530642396 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f966c02ff60 msgr2=0x7f966c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:14.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:14.650+0000 7f968730e700 1 --2- 172.21.15.146:0/2530642396 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f966c02ff60 0x7f966c032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f967c0096a0 tx=0x7f967c006b40).stop 2024-01-21T20:00:14.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:14.650+0000 7f968730e700 1 -- 172.21.15.146:0/2530642396 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96800f6510 msgr2=0x7f9680106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:14.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:14.650+0000 7f968730e700 1 --2- 172.21.15.146:0/2530642396 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96800f6510 0x7f9680106730 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f9670013510 tx=0x7f967000b040).stop 2024-01-21T20:00:14.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:14.650+0000 7f968730e700 1 -- 172.21.15.146:0/2530642396 shutdown_connections 2024-01-21T20:00:14.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:14.650+0000 7f968730e700 1 --2- 172.21.15.146:0/2530642396 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f966c02ff60 0x7f966c032420 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:14.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:14.650+0000 7f968730e700 1 --2- 172.21.15.146:0/2530642396 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96800f6510 0x7f9680106730 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:14.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:14.650+0000 7f968730e700 1 -- 172.21.15.146:0/2530642396 >> 172.21.15.146:0/2530642396 conn(0x7f96800f1c70 msgr2=0x7f96800f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:14.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:14.650+0000 7f968730e700 1 -- 172.21.15.146:0/2530642396 shutdown_connections 2024-01-21T20:00:14.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:14.650+0000 7f968730e700 1 -- 172.21.15.146:0/2530642396 wait complete. 2024-01-21T20:00:14.643 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:00:14.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:14 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:00:13.133106+0000 mgr.smithi053.fgsmol (mgr.14164) 33 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:15.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:15 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:00:14.649942+0000 mon.smithi053 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.146:0/2530642396' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:00:16.020 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:00:16.020 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:00:16.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:00:15.133517+0000 mgr.smithi053.fgsmol (mgr.14164) 34 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:17.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:17.211+0000 7f84cd304700 1 -- 172.21.15.146:0/3064500758 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84c80f5410 msgr2=0x7f84c80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:17.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:17.211+0000 7f84cd304700 1 --2- 172.21.15.146:0/3064500758 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84c80f5410 0x7f84c80f5830 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f84bc009fb0 tx=0x7f84bc009b30).stop 2024-01-21T20:00:17.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:17.211+0000 7f84cd304700 1 -- 172.21.15.146:0/3064500758 shutdown_connections 2024-01-21T20:00:17.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:17.211+0000 7f84cd304700 1 --2- 172.21.15.146:0/3064500758 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84c80f5410 0x7f84c80f5830 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:17.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:17.211+0000 7f84cd304700 1 -- 172.21.15.146:0/3064500758 >> 172.21.15.146:0/3064500758 conn(0x7f84c80f0b70 msgr2=0x7f84c80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:17.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:17.211+0000 7f84cd304700 1 -- 172.21.15.146:0/3064500758 shutdown_connections 2024-01-21T20:00:17.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:17.211+0000 7f84cd304700 1 -- 172.21.15.146:0/3064500758 wait complete. 2024-01-21T20:00:17.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:17.212+0000 7f84cd304700 1 Processor -- start 2024-01-21T20:00:17.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:17.212+0000 7f84cd304700 1 -- start start 2024-01-21T20:00:17.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:17.212+0000 7f84cd304700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84c80f5410 0x7f84c8109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:17.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:17.212+0000 7f84cd304700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c8060240 con 0x7f84c80f5410 2024-01-21T20:00:17.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:17.212+0000 7f84c6ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84c80f5410 0x7f84c8109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:17.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:17.212+0000 7f84c6ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84c80f5410 0x7f84c8109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:46878/0 (socket says 172.21.15.146:46878) 2024-01-21T20:00:17.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:17.212+0000 7f84c6ffd700 1 -- 172.21.15.146:0/2800007307 learned_addr learned my addr 172.21.15.146:0/2800007307 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:00:17.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:17.213+0000 7f84c6ffd700 1 -- 172.21.15.146:0/2800007307 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84bc005040 con 0x7f84c80f5410 2024-01-21T20:00:17.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:17.213+0000 7f84c6ffd700 1 --2- 172.21.15.146:0/2800007307 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84c80f5410 0x7f84c8109430 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f84bc014000 tx=0x7f84bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:17.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:17.213+0000 7f84af7fe700 1 -- 172.21.15.146:0/2800007307 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84bc0095f0 con 0x7f84c80f5410 2024-01-21T20:00:17.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:17.213+0000 7f84af7fe700 1 -- 172.21.15.146:0/2800007307 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f84bc007e70 con 0x7f84c80f5410 2024-01-21T20:00:17.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:17.213+0000 7f84cd304700 1 -- 172.21.15.146:0/2800007307 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84c8060440 con 0x7f84c80f5410 2024-01-21T20:00:17.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:17.213+0000 7f84af7fe700 1 -- 172.21.15.146:0/2800007307 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84bc014ea0 con 0x7f84c80f5410 2024-01-21T20:00:17.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:17.213+0000 7f84cd304700 1 -- 172.21.15.146:0/2800007307 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84c8109ae0 con 0x7f84c80f5410 2024-01-21T20:00:17.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:17.214+0000 7f84af7fe700 1 -- 172.21.15.146:0/2800007307 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f84bc025070 con 0x7f84c80f5410 2024-01-21T20:00:17.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:17.214+0000 7f84cd304700 1 -- 172.21.15.146:0/2800007307 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84c8045ad0 con 0x7f84c80f5410 2024-01-21T20:00:17.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:17.214+0000 7f84af7fe700 1 --2- 172.21.15.146:0/2800007307 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f84b002ff60 0x7f84b0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:17.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:17.214+0000 7f84af7fe700 1 -- 172.21.15.146:0/2800007307 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f84bc040920 con 0x7f84c80f5410 2024-01-21T20:00:17.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:17.214+0000 7f84c67fc700 1 --2- 172.21.15.146:0/2800007307 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f84b002ff60 0x7f84b0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:17.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:17.215+0000 7f84c67fc700 1 --2- 172.21.15.146:0/2800007307 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f84b002ff60 0x7f84b0032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f84b80096a0 tx=0x7f84b8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:17.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:17.219+0000 7f84af7fe700 1 -- 172.21.15.146:0/2800007307 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f84bc00f4e0 con 0x7f84c80f5410 2024-01-21T20:00:17.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:17.427+0000 7f84cd304700 1 -- 172.21.15.146:0/2800007307 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f84c80fae60 con 0x7f84c80f5410 2024-01-21T20:00:17.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:17.429+0000 7f84af7fe700 1 -- 172.21.15.146:0/2800007307 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f84bc00f300 con 0x7f84c80f5410 2024-01-21T20:00:17.420 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:00:17.420 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:00:17.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:17.430+0000 7f84cd304700 1 -- 172.21.15.146:0/2800007307 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f84b002ff60 msgr2=0x7f84b0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:17.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:17.430+0000 7f84cd304700 1 --2- 172.21.15.146:0/2800007307 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f84b002ff60 0x7f84b0032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f84b80096a0 tx=0x7f84b8006b40).stop 2024-01-21T20:00:17.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:17.430+0000 7f84cd304700 1 -- 172.21.15.146:0/2800007307 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84c80f5410 msgr2=0x7f84c8109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:17.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:17.430+0000 7f84cd304700 1 --2- 172.21.15.146:0/2800007307 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84c80f5410 0x7f84c8109430 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f84bc014000 tx=0x7f84bc00b040).stop 2024-01-21T20:00:17.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:17.430+0000 7f84cd304700 1 -- 172.21.15.146:0/2800007307 shutdown_connections 2024-01-21T20:00:17.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:17.430+0000 7f84cd304700 1 --2- 172.21.15.146:0/2800007307 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f84b002ff60 0x7f84b0032420 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:17.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:17.430+0000 7f84cd304700 1 --2- 172.21.15.146:0/2800007307 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84c80f5410 0x7f84c8109430 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:17.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:17.430+0000 7f84cd304700 1 -- 172.21.15.146:0/2800007307 >> 172.21.15.146:0/2800007307 conn(0x7f84c80f0b70 msgr2=0x7f84c80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:17.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:17.430+0000 7f84cd304700 1 -- 172.21.15.146:0/2800007307 shutdown_connections 2024-01-21T20:00:17.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:17.430+0000 7f84cd304700 1 -- 172.21.15.146:0/2800007307 wait complete. 2024-01-21T20:00:17.422 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:00:17.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:17 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:00:17.429247+0000 mon.smithi053 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.146:0/2800007307' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:00:18.838 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:00:18.838 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:00:18.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:18 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:00:17.134015+0000 mgr.smithi053.fgsmol (mgr.14164) 35 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:20.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:20.061+0000 7f88c160c700 1 -- 172.21.15.146:0/3512537471 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88bc0f62a0 msgr2=0x7f88bc0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:20.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:20.061+0000 7f88c160c700 1 --2- 172.21.15.146:0/3512537471 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88bc0f62a0 0x7f88bc0f66c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f88ac009fb0 tx=0x7f88ac009b30).stop 2024-01-21T20:00:20.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:20.061+0000 7f88c160c700 1 -- 172.21.15.146:0/3512537471 shutdown_connections 2024-01-21T20:00:20.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:20.061+0000 7f88c160c700 1 --2- 172.21.15.146:0/3512537471 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88bc0f62a0 0x7f88bc0f66c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:20.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:20.061+0000 7f88c160c700 1 -- 172.21.15.146:0/3512537471 >> 172.21.15.146:0/3512537471 conn(0x7f88bc0f1aa0 msgr2=0x7f88bc0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:20.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:20.061+0000 7f88c160c700 1 -- 172.21.15.146:0/3512537471 shutdown_connections 2024-01-21T20:00:20.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:20.061+0000 7f88c160c700 1 -- 172.21.15.146:0/3512537471 wait complete. 2024-01-21T20:00:20.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:20.061+0000 7f88c160c700 1 Processor -- start 2024-01-21T20:00:20.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:20.062+0000 7f88c160c700 1 -- start start 2024-01-21T20:00:20.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:20.062+0000 7f88c160c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88bc0f62a0 0x7f88bc1064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:20.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:20.062+0000 7f88c160c700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88bc1069a0 con 0x7f88bc0f62a0 2024-01-21T20:00:20.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:20.062+0000 7f88bbfff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88bc0f62a0 0x7f88bc1064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:20.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:20.062+0000 7f88bbfff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88bc0f62a0 0x7f88bc1064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:57666/0 (socket says 172.21.15.146:57666) 2024-01-21T20:00:20.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:20.062+0000 7f88bbfff700 1 -- 172.21.15.146:0/3746727426 learned_addr learned my addr 172.21.15.146:0/3746727426 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:00:20.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:20.063+0000 7f88bbfff700 1 -- 172.21.15.146:0/3746727426 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88ac005040 con 0x7f88bc0f62a0 2024-01-21T20:00:20.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:20.063+0000 7f88bbfff700 1 --2- 172.21.15.146:0/3746727426 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88bc0f62a0 0x7f88bc1064b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f88ac009830 tx=0x7f88ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:20.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:20.063+0000 7f88b8ff9700 1 -- 172.21.15.146:0/3746727426 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88ac009320 con 0x7f88bc0f62a0 2024-01-21T20:00:20.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:20.063+0000 7f88b8ff9700 1 -- 172.21.15.146:0/3746727426 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f88ac0136c0 con 0x7f88bc0f62a0 2024-01-21T20:00:20.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:20.063+0000 7f88c160c700 1 -- 172.21.15.146:0/3746727426 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88bc106ba0 con 0x7f88bc0f62a0 2024-01-21T20:00:20.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:20.063+0000 7f88b8ff9700 1 -- 172.21.15.146:0/3746727426 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88ac01ee90 con 0x7f88bc0f62a0 2024-01-21T20:00:20.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:20.063+0000 7f88c160c700 1 -- 172.21.15.146:0/3746727426 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88bc107030 con 0x7f88bc0f62a0 2024-01-21T20:00:20.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:20.064+0000 7f88b8ff9700 1 -- 172.21.15.146:0/3746727426 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f88ac01e420 con 0x7f88bc0f62a0 2024-01-21T20:00:20.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:20.064+0000 7f88c160c700 1 -- 172.21.15.146:0/3746727426 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88bc045ad0 con 0x7f88bc0f62a0 2024-01-21T20:00:20.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:20.064+0000 7f88b8ff9700 1 --2- 172.21.15.146:0/3746727426 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f88a402ff10 0x7f88a40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:20.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:20.064+0000 7f88b8ff9700 1 -- 172.21.15.146:0/3746727426 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f88ac040fe0 con 0x7f88bc0f62a0 2024-01-21T20:00:20.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:20.064+0000 7f88bb7fe700 1 --2- 172.21.15.146:0/3746727426 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f88a402ff10 0x7f88a40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:20.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:20.064+0000 7f88bb7fe700 1 --2- 172.21.15.146:0/3746727426 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f88a402ff10 0x7f88a40323d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f88b00096a0 tx=0x7f88b0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:20.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:20.069+0000 7f88b8ff9700 1 -- 172.21.15.146:0/3746727426 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f88ac0095f0 con 0x7f88bc0f62a0 2024-01-21T20:00:20.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:20.273+0000 7f88c160c700 1 -- 172.21.15.146:0/3746727426 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f88bc107ee0 con 0x7f88bc0f62a0 2024-01-21T20:00:20.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:20.274+0000 7f88b8ff9700 1 -- 172.21.15.146:0/3746727426 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f88ac02c2d0 con 0x7f88bc0f62a0 2024-01-21T20:00:20.266 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:00:20.266 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:00:20.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:20.276+0000 7f88c160c700 1 -- 172.21.15.146:0/3746727426 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f88a402ff10 msgr2=0x7f88a40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:20.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:20.276+0000 7f88c160c700 1 --2- 172.21.15.146:0/3746727426 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f88a402ff10 0x7f88a40323d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f88b00096a0 tx=0x7f88b0006b40).stop 2024-01-21T20:00:20.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:20.276+0000 7f88c160c700 1 -- 172.21.15.146:0/3746727426 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88bc0f62a0 msgr2=0x7f88bc1064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:20.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:20.276+0000 7f88c160c700 1 --2- 172.21.15.146:0/3746727426 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88bc0f62a0 0x7f88bc1064b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f88ac009830 tx=0x7f88ac00b040).stop 2024-01-21T20:00:20.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:20.276+0000 7f88c160c700 1 -- 172.21.15.146:0/3746727426 shutdown_connections 2024-01-21T20:00:20.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:20.276+0000 7f88c160c700 1 --2- 172.21.15.146:0/3746727426 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f88a402ff10 0x7f88a40323d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:20.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:20.276+0000 7f88c160c700 1 --2- 172.21.15.146:0/3746727426 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88bc0f62a0 0x7f88bc1064b0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:20.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:20.276+0000 7f88c160c700 1 -- 172.21.15.146:0/3746727426 >> 172.21.15.146:0/3746727426 conn(0x7f88bc0f1aa0 msgr2=0x7f88bc0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:20.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:20.276+0000 7f88c160c700 1 -- 172.21.15.146:0/3746727426 shutdown_connections 2024-01-21T20:00:20.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:20.276+0000 7f88c160c700 1 -- 172.21.15.146:0/3746727426 wait complete. 2024-01-21T20:00:20.269 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:00:20.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:20 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:00:19.134311+0000 mgr.smithi053.fgsmol (mgr.14164) 36 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:20.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:20 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:00:20.275612+0000 mon.smithi053 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.146:0/3746727426' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:00:21.720 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:00:21.720 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:00:22.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:00:21.134730+0000 mgr.smithi053.fgsmol (mgr.14164) 37 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:22.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:22.955+0000 7fb8cd371700 1 -- 172.21.15.146:0/960027522 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8c80f51a0 msgr2=0x7fb8c80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:22.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:22.955+0000 7fb8cd371700 1 --2- 172.21.15.146:0/960027522 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8c80f51a0 0x7fb8c80f55c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fb8b8009fb0 tx=0x7fb8b8009b30).stop 2024-01-21T20:00:22.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:22.955+0000 7fb8cd371700 1 -- 172.21.15.146:0/960027522 shutdown_connections 2024-01-21T20:00:22.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:22.955+0000 7fb8cd371700 1 --2- 172.21.15.146:0/960027522 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8c80f51a0 0x7fb8c80f55c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:22.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:22.955+0000 7fb8cd371700 1 -- 172.21.15.146:0/960027522 >> 172.21.15.146:0/960027522 conn(0x7fb8c80f09a0 msgr2=0x7fb8c80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:22.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:22.955+0000 7fb8cd371700 1 -- 172.21.15.146:0/960027522 shutdown_connections 2024-01-21T20:00:22.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:22.955+0000 7fb8cd371700 1 -- 172.21.15.146:0/960027522 wait complete. 2024-01-21T20:00:22.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:22.955+0000 7fb8cd371700 1 Processor -- start 2024-01-21T20:00:22.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:22.955+0000 7fb8cd371700 1 -- start start 2024-01-21T20:00:22.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:22.956+0000 7fb8cd371700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8c80f51a0 0x7fb8c81091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:22.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:22.956+0000 7fb8cd371700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c8060240 con 0x7fb8c80f51a0 2024-01-21T20:00:22.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:22.956+0000 7fb8c7fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8c80f51a0 0x7fb8c81091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:22.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:22.956+0000 7fb8c7fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8c80f51a0 0x7fb8c81091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:57684/0 (socket says 172.21.15.146:57684) 2024-01-21T20:00:22.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:22.956+0000 7fb8c7fff700 1 -- 172.21.15.146:0/3362312380 learned_addr learned my addr 172.21.15.146:0/3362312380 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:00:22.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:22.956+0000 7fb8c7fff700 1 -- 172.21.15.146:0/3362312380 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b8005040 con 0x7fb8c80f51a0 2024-01-21T20:00:22.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:22.956+0000 7fb8c7fff700 1 --2- 172.21.15.146:0/3362312380 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8c80f51a0 0x7fb8c81091b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fb8b8013510 tx=0x7fb8b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:22.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:22.956+0000 7fb8c4ff9700 1 -- 172.21.15.146:0/3362312380 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8b8009320 con 0x7fb8c80f51a0 2024-01-21T20:00:22.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:22.956+0000 7fb8c4ff9700 1 -- 172.21.15.146:0/3362312380 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb8b8013f00 con 0x7fb8c80f51a0 2024-01-21T20:00:22.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:22.956+0000 7fb8c4ff9700 1 -- 172.21.15.146:0/3362312380 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8b8014dc0 con 0x7fb8c80f51a0 2024-01-21T20:00:22.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:22.956+0000 7fb8cd371700 1 -- 172.21.15.146:0/3362312380 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8c8060440 con 0x7fb8c80f51a0 2024-01-21T20:00:22.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:22.957+0000 7fb8cd371700 1 -- 172.21.15.146:0/3362312380 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8c8109860 con 0x7fb8c80f51a0 2024-01-21T20:00:22.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:22.957+0000 7fb8c4ff9700 1 -- 172.21.15.146:0/3362312380 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb8b80095f0 con 0x7fb8c80f51a0 2024-01-21T20:00:22.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:22.957+0000 7fb8cd371700 1 -- 172.21.15.146:0/3362312380 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8c8045ad0 con 0x7fb8c80f51a0 2024-01-21T20:00:22.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:22.957+0000 7fb8c4ff9700 1 --2- 172.21.15.146:0/3362312380 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb8b002ff60 0x7fb8b0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:22.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:22.957+0000 7fb8c4ff9700 1 -- 172.21.15.146:0/3362312380 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb8b8041040 con 0x7fb8c80f51a0 2024-01-21T20:00:22.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:22.958+0000 7fb8c77fe700 1 --2- 172.21.15.146:0/3362312380 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb8b002ff60 0x7fb8b0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:22.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:22.958+0000 7fb8c77fe700 1 --2- 172.21.15.146:0/3362312380 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb8b002ff60 0x7fb8b0032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fb8bc0096a0 tx=0x7fb8bc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:22.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:22.962+0000 7fb8c4ff9700 1 -- 172.21.15.146:0/3362312380 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb8b8014420 con 0x7fb8c80f51a0 2024-01-21T20:00:23.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:23.165+0000 7fb8cd371700 1 -- 172.21.15.146:0/3362312380 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb8c810a080 con 0x7fb8c80f51a0 2024-01-21T20:00:23.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:23.165+0000 7fb8c4ff9700 1 -- 172.21.15.146:0/3362312380 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb8b8010230 con 0x7fb8c80f51a0 2024-01-21T20:00:23.156 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:00:23.156 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:00:23.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:23.167+0000 7fb8cd371700 1 -- 172.21.15.146:0/3362312380 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb8b002ff60 msgr2=0x7fb8b0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:23.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:23.167+0000 7fb8cd371700 1 --2- 172.21.15.146:0/3362312380 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb8b002ff60 0x7fb8b0032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fb8bc0096a0 tx=0x7fb8bc006b40).stop 2024-01-21T20:00:23.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:23.167+0000 7fb8cd371700 1 -- 172.21.15.146:0/3362312380 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8c80f51a0 msgr2=0x7fb8c81091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:23.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:23.167+0000 7fb8cd371700 1 --2- 172.21.15.146:0/3362312380 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8c80f51a0 0x7fb8c81091b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fb8b8013510 tx=0x7fb8b800b040).stop 2024-01-21T20:00:23.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:23.167+0000 7fb8cd371700 1 -- 172.21.15.146:0/3362312380 shutdown_connections 2024-01-21T20:00:23.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:23.167+0000 7fb8cd371700 1 --2- 172.21.15.146:0/3362312380 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb8b002ff60 0x7fb8b0032420 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:23.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:23.167+0000 7fb8cd371700 1 --2- 172.21.15.146:0/3362312380 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8c80f51a0 0x7fb8c81091b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:23.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:23.167+0000 7fb8cd371700 1 -- 172.21.15.146:0/3362312380 >> 172.21.15.146:0/3362312380 conn(0x7fb8c80f09a0 msgr2=0x7fb8c80f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:23.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:23.167+0000 7fb8cd371700 1 -- 172.21.15.146:0/3362312380 shutdown_connections 2024-01-21T20:00:23.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:23.167+0000 7fb8cd371700 1 -- 172.21.15.146:0/3362312380 wait complete. 2024-01-21T20:00:23.159 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:00:23.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:23 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:00:23.165675+0000 mon.smithi053 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.146:0/3362312380' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:00:24.679 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:00:24.680 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:00:24.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:24 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:00:23.135155+0000 mgr.smithi053.fgsmol (mgr.14164) 38 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:25.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:25.933+0000 7f9bb4bb2700 1 -- 172.21.15.146:0/1712133182 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bb00f6510 msgr2=0x7f9bb00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:25.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:25.933+0000 7f9bb4bb2700 1 --2- 172.21.15.146:0/1712133182 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bb00f6510 0x7f9bb00f6930 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f9ba4009fb0 tx=0x7f9ba4009b30).stop 2024-01-21T20:00:25.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:25.933+0000 7f9bb4bb2700 1 -- 172.21.15.146:0/1712133182 shutdown_connections 2024-01-21T20:00:25.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:25.933+0000 7f9bb4bb2700 1 --2- 172.21.15.146:0/1712133182 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bb00f6510 0x7f9bb00f6930 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:25.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:25.933+0000 7f9bb4bb2700 1 -- 172.21.15.146:0/1712133182 >> 172.21.15.146:0/1712133182 conn(0x7f9bb00f1c70 msgr2=0x7f9bb00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:25.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:25.933+0000 7f9bb4bb2700 1 -- 172.21.15.146:0/1712133182 shutdown_connections 2024-01-21T20:00:25.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:25.933+0000 7f9bb4bb2700 1 -- 172.21.15.146:0/1712133182 wait complete. 2024-01-21T20:00:25.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:25.933+0000 7f9bb4bb2700 1 Processor -- start 2024-01-21T20:00:25.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:25.934+0000 7f9bb4bb2700 1 -- start start 2024-01-21T20:00:25.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:25.934+0000 7f9bb4bb2700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bb00f6510 0x7f9bb0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:25.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:25.934+0000 7f9bb4bb2700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb0106c20 con 0x7f9bb00f6510 2024-01-21T20:00:25.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:25.934+0000 7f9bae59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bb00f6510 0x7f9bb0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:25.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:25.934+0000 7f9bae59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bb00f6510 0x7f9bb0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:57710/0 (socket says 172.21.15.146:57710) 2024-01-21T20:00:25.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:25.934+0000 7f9bae59c700 1 -- 172.21.15.146:0/997158632 learned_addr learned my addr 172.21.15.146:0/997158632 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:00:25.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:25.934+0000 7f9bae59c700 1 -- 172.21.15.146:0/997158632 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ba4005040 con 0x7f9bb00f6510 2024-01-21T20:00:25.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:25.934+0000 7f9bae59c700 1 --2- 172.21.15.146:0/997158632 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bb00f6510 0x7f9bb0106730 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f9ba4013510 tx=0x7f9ba400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:25.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:25.935+0000 7f9b9effd700 1 -- 172.21.15.146:0/997158632 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ba4009320 con 0x7f9bb00f6510 2024-01-21T20:00:25.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:25.935+0000 7f9bb4bb2700 1 -- 172.21.15.146:0/997158632 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bb0106e20 con 0x7f9bb00f6510 2024-01-21T20:00:25.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:25.935+0000 7f9b9effd700 1 -- 172.21.15.146:0/997158632 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9ba4013f00 con 0x7f9bb00f6510 2024-01-21T20:00:25.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:25.935+0000 7f9b9effd700 1 -- 172.21.15.146:0/997158632 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ba4014d90 con 0x7f9bb00f6510 2024-01-21T20:00:25.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:25.935+0000 7f9bb4bb2700 1 -- 172.21.15.146:0/997158632 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bb01072b0 con 0x7f9bb00f6510 2024-01-21T20:00:25.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:25.936+0000 7f9b9effd700 1 -- 172.21.15.146:0/997158632 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9ba40095f0 con 0x7f9bb00f6510 2024-01-21T20:00:25.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:25.936+0000 7f9bb4bb2700 1 -- 172.21.15.146:0/997158632 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bb0045ad0 con 0x7f9bb00f6510 2024-01-21T20:00:25.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:25.936+0000 7f9b9effd700 1 --2- 172.21.15.146:0/997158632 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f9b9802ff60 0x7f9b98032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:25.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:25.936+0000 7f9b9effd700 1 -- 172.21.15.146:0/997158632 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9ba4041000 con 0x7f9bb00f6510 2024-01-21T20:00:25.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:25.936+0000 7f9badd9b700 1 --2- 172.21.15.146:0/997158632 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f9b9802ff60 0x7f9b98032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:25.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:25.937+0000 7f9badd9b700 1 --2- 172.21.15.146:0/997158632 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f9b9802ff60 0x7f9b98032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f9ba00096a0 tx=0x7f9ba0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:25.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:25.941+0000 7f9b9effd700 1 -- 172.21.15.146:0/997158632 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9ba4014420 con 0x7f9bb00f6510 2024-01-21T20:00:26.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:26.143+0000 7f9bb4bb2700 1 -- 172.21.15.146:0/997158632 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9bb00fbf60 con 0x7f9bb00f6510 2024-01-21T20:00:26.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:26.144+0000 7f9b9effd700 1 -- 172.21.15.146:0/997158632 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9ba401d7f0 con 0x7f9bb00f6510 2024-01-21T20:00:26.135 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:00:26.135 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:00:26.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:26.145+0000 7f9bb4bb2700 1 -- 172.21.15.146:0/997158632 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f9b9802ff60 msgr2=0x7f9b98032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:26.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:26.145+0000 7f9bb4bb2700 1 --2- 172.21.15.146:0/997158632 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f9b9802ff60 0x7f9b98032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f9ba00096a0 tx=0x7f9ba0006b40).stop 2024-01-21T20:00:26.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:26.145+0000 7f9bb4bb2700 1 -- 172.21.15.146:0/997158632 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bb00f6510 msgr2=0x7f9bb0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:26.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:26.145+0000 7f9bb4bb2700 1 --2- 172.21.15.146:0/997158632 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bb00f6510 0x7f9bb0106730 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f9ba4013510 tx=0x7f9ba400b040).stop 2024-01-21T20:00:26.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:26.146+0000 7f9bb4bb2700 1 -- 172.21.15.146:0/997158632 shutdown_connections 2024-01-21T20:00:26.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:26.146+0000 7f9bb4bb2700 1 --2- 172.21.15.146:0/997158632 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f9b9802ff60 0x7f9b98032420 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:26.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:26.146+0000 7f9bb4bb2700 1 --2- 172.21.15.146:0/997158632 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bb00f6510 0x7f9bb0106730 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:26.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:26.146+0000 7f9bb4bb2700 1 -- 172.21.15.146:0/997158632 >> 172.21.15.146:0/997158632 conn(0x7f9bb00f1c70 msgr2=0x7f9bb00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:26.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:26.146+0000 7f9bb4bb2700 1 -- 172.21.15.146:0/997158632 shutdown_connections 2024-01-21T20:00:26.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:26.146+0000 7f9bb4bb2700 1 -- 172.21.15.146:0/997158632 wait complete. 2024-01-21T20:00:26.140 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:00:26.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:26 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:00:25.135603+0000 mgr.smithi053.fgsmol (mgr.14164) 39 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:26.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:26 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:00:26.144199+0000 mon.smithi053 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.146:0/997158632' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:00:27.565 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:00:27.566 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:00:28.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:28.759+0000 7fdaa7e39700 1 -- 172.21.15.146:0/1661192087 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdaa00601e0 msgr2=0x7fdaa00f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:28.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:28.759+0000 7fdaa7e39700 1 --2- 172.21.15.146:0/1661192087 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdaa00601e0 0x7fdaa00f9190 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fda90009fb0 tx=0x7fda90009b30).stop 2024-01-21T20:00:28.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:28.759+0000 7fdaa7e39700 1 -- 172.21.15.146:0/1661192087 shutdown_connections 2024-01-21T20:00:28.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:28.759+0000 7fdaa7e39700 1 --2- 172.21.15.146:0/1661192087 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdaa00601e0 0x7fdaa00f9190 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:28.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:28.759+0000 7fdaa7e39700 1 -- 172.21.15.146:0/1661192087 >> 172.21.15.146:0/1661192087 conn(0x7fdaa00f0b70 msgr2=0x7fdaa00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:28.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:28.759+0000 7fdaa7e39700 1 -- 172.21.15.146:0/1661192087 shutdown_connections 2024-01-21T20:00:28.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:28.759+0000 7fdaa7e39700 1 -- 172.21.15.146:0/1661192087 wait complete. 2024-01-21T20:00:28.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:28.760+0000 7fdaa7e39700 1 Processor -- start 2024-01-21T20:00:28.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:28.760+0000 7fdaa7e39700 1 -- start start 2024-01-21T20:00:28.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:28.760+0000 7fdaa7e39700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdaa00601e0 0x7fdaa01066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:28.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:28.760+0000 7fdaa7e39700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa0106bd0 con 0x7fdaa00601e0 2024-01-21T20:00:28.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:28.760+0000 7fdaa5bd5700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdaa00601e0 0x7fdaa01066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:28.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:28.760+0000 7fdaa5bd5700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdaa00601e0 0x7fdaa01066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:42886/0 (socket says 172.21.15.146:42886) 2024-01-21T20:00:28.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:28.760+0000 7fdaa5bd5700 1 -- 172.21.15.146:0/746139022 learned_addr learned my addr 172.21.15.146:0/746139022 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:00:28.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:28.760+0000 7fdaa5bd5700 1 -- 172.21.15.146:0/746139022 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda90005040 con 0x7fdaa00601e0 2024-01-21T20:00:28.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:28.760+0000 7fdaa5bd5700 1 --2- 172.21.15.146:0/746139022 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdaa00601e0 0x7fdaa01066e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fda900079e0 tx=0x7fda90009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:28.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:28.761+0000 7fda967fc700 1 -- 172.21.15.146:0/746139022 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda90018070 con 0x7fdaa00601e0 2024-01-21T20:00:28.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:28.761+0000 7fdaa7e39700 1 -- 172.21.15.146:0/746139022 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaa0106dd0 con 0x7fdaa00601e0 2024-01-21T20:00:28.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:28.761+0000 7fda967fc700 1 -- 172.21.15.146:0/746139022 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fda90013410 con 0x7fdaa00601e0 2024-01-21T20:00:28.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:28.761+0000 7fda967fc700 1 -- 172.21.15.146:0/746139022 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda9001c3c0 con 0x7fdaa00601e0 2024-01-21T20:00:28.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:28.761+0000 7fdaa7e39700 1 -- 172.21.15.146:0/746139022 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaa0107260 con 0x7fdaa00601e0 2024-01-21T20:00:28.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:28.761+0000 7fda967fc700 1 -- 172.21.15.146:0/746139022 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fda9001d420 con 0x7fdaa00601e0 2024-01-21T20:00:28.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:28.761+0000 7fdaa7e39700 1 -- 172.21.15.146:0/746139022 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaa0045ad0 con 0x7fdaa00601e0 2024-01-21T20:00:28.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:28.762+0000 7fda967fc700 1 --2- 172.21.15.146:0/746139022 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fda8c02ff60 0x7fda8c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:28.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:28.762+0000 7fda967fc700 1 -- 172.21.15.146:0/746139022 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fda90041460 con 0x7fdaa00601e0 2024-01-21T20:00:28.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:28.762+0000 7fdaa53d4700 1 --2- 172.21.15.146:0/746139022 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fda8c02ff60 0x7fda8c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:28.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:28.762+0000 7fdaa53d4700 1 --2- 172.21.15.146:0/746139022 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fda8c02ff60 0x7fda8c032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fda9c0096a0 tx=0x7fda9c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:28.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:28.767+0000 7fda967fc700 1 -- 172.21.15.146:0/746139022 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fda9001d7e0 con 0x7fdaa00601e0 2024-01-21T20:00:28.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:28 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:00:27.135985+0000 mgr.smithi053.fgsmol (mgr.14164) 40 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:28.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:28.972+0000 7fdaa7e39700 1 -- 172.21.15.146:0/746139022 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdaa01080b0 con 0x7fdaa00601e0 2024-01-21T20:00:28.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:28.973+0000 7fda967fc700 1 -- 172.21.15.146:0/746139022 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fda9001b370 con 0x7fdaa00601e0 2024-01-21T20:00:28.964 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:00:28.964 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:00:28.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:28.974+0000 7fdaa7e39700 1 -- 172.21.15.146:0/746139022 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fda8c02ff60 msgr2=0x7fda8c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:28.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:28.974+0000 7fdaa7e39700 1 --2- 172.21.15.146:0/746139022 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fda8c02ff60 0x7fda8c032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fda9c0096a0 tx=0x7fda9c006b40).stop 2024-01-21T20:00:28.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:28.974+0000 7fdaa7e39700 1 -- 172.21.15.146:0/746139022 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdaa00601e0 msgr2=0x7fdaa01066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:28.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:28.974+0000 7fdaa7e39700 1 --2- 172.21.15.146:0/746139022 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdaa00601e0 0x7fdaa01066e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fda900079e0 tx=0x7fda90009b30).stop 2024-01-21T20:00:28.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:28.975+0000 7fdaa7e39700 1 -- 172.21.15.146:0/746139022 shutdown_connections 2024-01-21T20:00:28.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:28.975+0000 7fdaa7e39700 1 --2- 172.21.15.146:0/746139022 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fda8c02ff60 0x7fda8c032420 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:28.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:28.975+0000 7fdaa7e39700 1 --2- 172.21.15.146:0/746139022 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdaa00601e0 0x7fdaa01066e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:28.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:28.975+0000 7fdaa7e39700 1 -- 172.21.15.146:0/746139022 >> 172.21.15.146:0/746139022 conn(0x7fdaa00f0b70 msgr2=0x7fdaa00f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:28.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:28.975+0000 7fdaa7e39700 1 -- 172.21.15.146:0/746139022 shutdown_connections 2024-01-21T20:00:28.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:28.975+0000 7fdaa7e39700 1 -- 172.21.15.146:0/746139022 wait complete. 2024-01-21T20:00:28.966 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:00:29.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:29 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:00:28.973098+0000 mon.smithi053 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.146:0/746139022' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:00:30.396 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:00:30.396 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:00:30.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:30 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:00:29.136371+0000 mgr.smithi053.fgsmol (mgr.14164) 41 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:31.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:31.637+0000 7f28136e9700 1 -- 172.21.15.146:0/1976030098 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f280c060280 msgr2=0x7f280c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:31.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:31.637+0000 7f28136e9700 1 --2- 172.21.15.146:0/1976030098 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f280c060280 0x7f280c0f90f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f27fc009fb0 tx=0x7f27fc009b30).stop 2024-01-21T20:00:31.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:31.637+0000 7f28136e9700 1 -- 172.21.15.146:0/1976030098 shutdown_connections 2024-01-21T20:00:31.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:31.637+0000 7f28136e9700 1 --2- 172.21.15.146:0/1976030098 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f280c060280 0x7f280c0f90f0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:31.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:31.637+0000 7f28136e9700 1 -- 172.21.15.146:0/1976030098 >> 172.21.15.146:0/1976030098 conn(0x7f280c0f0b70 msgr2=0x7f280c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:31.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:31.637+0000 7f28136e9700 1 -- 172.21.15.146:0/1976030098 shutdown_connections 2024-01-21T20:00:31.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:31.637+0000 7f28136e9700 1 -- 172.21.15.146:0/1976030098 wait complete. 2024-01-21T20:00:31.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:31.638+0000 7f28136e9700 1 Processor -- start 2024-01-21T20:00:31.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:31.638+0000 7f28136e9700 1 -- start start 2024-01-21T20:00:31.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:31.638+0000 7f28136e9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f280c060280 0x7f280c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:31.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:31.638+0000 7f28136e9700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f280c106bd0 con 0x7f280c060280 2024-01-21T20:00:31.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:31.638+0000 7f2811485700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f280c060280 0x7f280c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:31.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:31.638+0000 7f2811485700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f280c060280 0x7f280c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:42894/0 (socket says 172.21.15.146:42894) 2024-01-21T20:00:31.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:31.638+0000 7f2811485700 1 -- 172.21.15.146:0/2862039841 learned_addr learned my addr 172.21.15.146:0/2862039841 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:00:31.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:31.638+0000 7f2811485700 1 -- 172.21.15.146:0/2862039841 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27fc005040 con 0x7f280c060280 2024-01-21T20:00:31.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:31.638+0000 7f2811485700 1 --2- 172.21.15.146:0/2862039841 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f280c060280 0x7f280c1066e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f27fc013510 tx=0x7f27fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:31.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:31.639+0000 7f2801ffb700 1 -- 172.21.15.146:0/2862039841 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27fc009320 con 0x7f280c060280 2024-01-21T20:00:31.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:31.639+0000 7f28136e9700 1 -- 172.21.15.146:0/2862039841 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f280c106dd0 con 0x7f280c060280 2024-01-21T20:00:31.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:31.639+0000 7f2801ffb700 1 -- 172.21.15.146:0/2862039841 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f27fc013f00 con 0x7f280c060280 2024-01-21T20:00:31.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:31.639+0000 7f2801ffb700 1 -- 172.21.15.146:0/2862039841 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27fc014dc0 con 0x7f280c060280 2024-01-21T20:00:31.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:31.639+0000 7f28136e9700 1 -- 172.21.15.146:0/2862039841 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f280c107260 con 0x7f280c060280 2024-01-21T20:00:31.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:31.640+0000 7f28136e9700 1 -- 172.21.15.146:0/2862039841 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f280c045ad0 con 0x7f280c060280 2024-01-21T20:00:31.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:31.640+0000 7f2801ffb700 1 -- 172.21.15.146:0/2862039841 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f27fc0095f0 con 0x7f280c060280 2024-01-21T20:00:31.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:31.640+0000 7f2801ffb700 1 --2- 172.21.15.146:0/2862039841 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f27f802ff60 0x7f27f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:31.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:31.640+0000 7f2801ffb700 1 -- 172.21.15.146:0/2862039841 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f27fc041020 con 0x7f280c060280 2024-01-21T20:00:31.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:31.640+0000 7f2810c84700 1 --2- 172.21.15.146:0/2862039841 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f27f802ff60 0x7f27f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:31.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:31.641+0000 7f2810c84700 1 --2- 172.21.15.146:0/2862039841 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f27f802ff60 0x7f27f8032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f28080096a0 tx=0x7f2808006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:31.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:31.645+0000 7f2801ffb700 1 -- 172.21.15.146:0/2862039841 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f27fc014420 con 0x7f280c060280 2024-01-21T20:00:31.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:31.851+0000 7f28136e9700 1 -- 172.21.15.146:0/2862039841 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f280c1080b0 con 0x7f280c060280 2024-01-21T20:00:31.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:31.852+0000 7f2801ffb700 1 -- 172.21.15.146:0/2862039841 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f27fc01c920 con 0x7f280c060280 2024-01-21T20:00:31.842 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:00:31.842 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:00:31.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:31.853+0000 7f28136e9700 1 -- 172.21.15.146:0/2862039841 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f27f802ff60 msgr2=0x7f27f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:31.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:31.853+0000 7f28136e9700 1 --2- 172.21.15.146:0/2862039841 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f27f802ff60 0x7f27f8032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f28080096a0 tx=0x7f2808006b40).stop 2024-01-21T20:00:31.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:31.853+0000 7f28136e9700 1 -- 172.21.15.146:0/2862039841 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f280c060280 msgr2=0x7f280c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:31.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:31.853+0000 7f28136e9700 1 --2- 172.21.15.146:0/2862039841 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f280c060280 0x7f280c1066e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f27fc013510 tx=0x7f27fc00b040).stop 2024-01-21T20:00:31.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:31.853+0000 7f28136e9700 1 -- 172.21.15.146:0/2862039841 shutdown_connections 2024-01-21T20:00:31.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:31.853+0000 7f28136e9700 1 --2- 172.21.15.146:0/2862039841 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f27f802ff60 0x7f27f8032420 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:31.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:31.853+0000 7f28136e9700 1 --2- 172.21.15.146:0/2862039841 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f280c060280 0x7f280c1066e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:31.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:31.853+0000 7f28136e9700 1 -- 172.21.15.146:0/2862039841 >> 172.21.15.146:0/2862039841 conn(0x7f280c0f0b70 msgr2=0x7f280c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:31.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:31.853+0000 7f28136e9700 1 -- 172.21.15.146:0/2862039841 shutdown_connections 2024-01-21T20:00:31.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:31.853+0000 7f28136e9700 1 -- 172.21.15.146:0/2862039841 wait complete. 2024-01-21T20:00:31.845 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:00:32.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:32 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:00:31.136670+0000 mgr.smithi053.fgsmol (mgr.14164) 42 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:32.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:32 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:00:31.852071+0000 mon.smithi053 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.146:0/2862039841' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:00:33.267 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:00:33.267 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:00:34.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:34 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:00:33.136983+0000 mgr.smithi053.fgsmol (mgr.14164) 43 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:35.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:35.913+0000 7f992e7d1700 1 -- 172.21.15.146:0/287257097 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99280f5410 msgr2=0x7f99280f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:35.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:35.913+0000 7f992e7d1700 1 --2- 172.21.15.146:0/287257097 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99280f5410 0x7f99280f5830 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f9918009fb0 tx=0x7f9918009b30).stop 2024-01-21T20:00:35.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:35.913+0000 7f992e7d1700 1 -- 172.21.15.146:0/287257097 shutdown_connections 2024-01-21T20:00:35.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:35.913+0000 7f992e7d1700 1 --2- 172.21.15.146:0/287257097 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99280f5410 0x7f99280f5830 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:35.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:35.913+0000 7f992e7d1700 1 -- 172.21.15.146:0/287257097 >> 172.21.15.146:0/287257097 conn(0x7f99280f0b70 msgr2=0x7f99280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:35.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:35.913+0000 7f992e7d1700 1 -- 172.21.15.146:0/287257097 shutdown_connections 2024-01-21T20:00:35.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:35.913+0000 7f992e7d1700 1 -- 172.21.15.146:0/287257097 wait complete. 2024-01-21T20:00:35.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:35.914+0000 7f992e7d1700 1 Processor -- start 2024-01-21T20:00:35.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:35.914+0000 7f992e7d1700 1 -- start start 2024-01-21T20:00:35.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:35.914+0000 7f992e7d1700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99280fd3e0 0x7f99280fb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:35.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:35.914+0000 7f992e7d1700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99280fd800 con 0x7f99280fd3e0 2024-01-21T20:00:35.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:35.914+0000 7f9927fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99280fd3e0 0x7f99280fb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:35.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:35.914+0000 7f9927fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99280fd3e0 0x7f99280fb8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:42910/0 (socket says 172.21.15.146:42910) 2024-01-21T20:00:35.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:35.914+0000 7f9927fff700 1 -- 172.21.15.146:0/3384197135 learned_addr learned my addr 172.21.15.146:0/3384197135 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:00:35.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:35.915+0000 7f9927fff700 1 -- 172.21.15.146:0/3384197135 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9918005040 con 0x7f99280fd3e0 2024-01-21T20:00:35.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:35.915+0000 7f9927fff700 1 --2- 172.21.15.146:0/3384197135 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99280fd3e0 0x7f99280fb8a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f9918009f80 tx=0x7f9918009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:35.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:35.915+0000 7f9924ff9700 1 -- 172.21.15.146:0/3384197135 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99180136c0 con 0x7f99280fd3e0 2024-01-21T20:00:35.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:35.915+0000 7f992e7d1700 1 -- 172.21.15.146:0/3384197135 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99280fbd90 con 0x7f99280fd3e0 2024-01-21T20:00:35.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:35.915+0000 7f9924ff9700 1 -- 172.21.15.146:0/3384197135 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9918013e70 con 0x7f99280fd3e0 2024-01-21T20:00:35.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:35.915+0000 7f9924ff9700 1 -- 172.21.15.146:0/3384197135 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9918014420 con 0x7f99280fd3e0 2024-01-21T20:00:35.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:35.915+0000 7f992e7d1700 1 -- 172.21.15.146:0/3384197135 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99280fc280 con 0x7f99280fd3e0 2024-01-21T20:00:35.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:35.916+0000 7f9924ff9700 1 -- 172.21.15.146:0/3384197135 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9918025060 con 0x7f99280fd3e0 2024-01-21T20:00:35.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:35.916+0000 7f992e7d1700 1 -- 172.21.15.146:0/3384197135 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9928045ad0 con 0x7f99280fd3e0 2024-01-21T20:00:35.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:35.916+0000 7f9924ff9700 1 --2- 172.21.15.146:0/3384197135 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f991002ff60 0x7f9910032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:35.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:35.916+0000 7f9924ff9700 1 -- 172.21.15.146:0/3384197135 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9918040ec0 con 0x7f99280fd3e0 2024-01-21T20:00:35.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:35.916+0000 7f99277fe700 1 --2- 172.21.15.146:0/3384197135 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f991002ff60 0x7f9910032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:35.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:35.917+0000 7f99277fe700 1 --2- 172.21.15.146:0/3384197135 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f991002ff60 0x7f9910032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f991c0096a0 tx=0x7f991c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:35.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:35.921+0000 7f9924ff9700 1 -- 172.21.15.146:0/3384197135 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f991801cd10 con 0x7f99280fd3e0 2024-01-21T20:00:36.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:36.125+0000 7f992e7d1700 1 -- 172.21.15.146:0/3384197135 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f992810ca40 con 0x7f99280fd3e0 2024-01-21T20:00:36.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:36.126+0000 7f9924ff9700 1 -- 172.21.15.146:0/3384197135 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f991802d2d0 con 0x7f99280fd3e0 2024-01-21T20:00:36.117 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:00:36.117 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:00:36.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:36.127+0000 7f992e7d1700 1 -- 172.21.15.146:0/3384197135 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f991002ff60 msgr2=0x7f9910032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:36.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:36.127+0000 7f992e7d1700 1 --2- 172.21.15.146:0/3384197135 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f991002ff60 0x7f9910032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f991c0096a0 tx=0x7f991c006b40).stop 2024-01-21T20:00:36.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:36.127+0000 7f992e7d1700 1 -- 172.21.15.146:0/3384197135 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99280fd3e0 msgr2=0x7f99280fb8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:36.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:36.127+0000 7f992e7d1700 1 --2- 172.21.15.146:0/3384197135 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99280fd3e0 0x7f99280fb8a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f9918009f80 tx=0x7f9918009b30).stop 2024-01-21T20:00:36.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:36.127+0000 7f992e7d1700 1 -- 172.21.15.146:0/3384197135 shutdown_connections 2024-01-21T20:00:36.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:36.127+0000 7f992e7d1700 1 --2- 172.21.15.146:0/3384197135 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f991002ff60 0x7f9910032420 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:36.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:36.127+0000 7f992e7d1700 1 --2- 172.21.15.146:0/3384197135 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99280fd3e0 0x7f99280fb8a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:36.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:36.127+0000 7f992e7d1700 1 -- 172.21.15.146:0/3384197135 >> 172.21.15.146:0/3384197135 conn(0x7f99280f0b70 msgr2=0x7f992805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:36.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:36.127+0000 7f992e7d1700 1 -- 172.21.15.146:0/3384197135 shutdown_connections 2024-01-21T20:00:36.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:36.127+0000 7f992e7d1700 1 -- 172.21.15.146:0/3384197135 wait complete. 2024-01-21T20:00:36.119 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:00:36.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:36 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:00:35.137309+0000 mgr.smithi053.fgsmol (mgr.14164) 44 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:36.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:36 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:00:36.126841+0000 mon.smithi053 (mon.0) 139 : audit [DBG] from='client.? 172.21.15.146:0/3384197135' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:00:37.527 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:00:37.528 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:00:38.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:38.824+0000 7f827ff03700 1 -- 172.21.15.146:0/3376734931 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82780f5410 msgr2=0x7f82780f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:38.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:38.824+0000 7f827ff03700 1 --2- 172.21.15.146:0/3376734931 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82780f5410 0x7f82780f5830 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f8268009fb0 tx=0x7f8268009b30).stop 2024-01-21T20:00:38.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:38.824+0000 7f827ff03700 1 -- 172.21.15.146:0/3376734931 shutdown_connections 2024-01-21T20:00:38.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:38.824+0000 7f827ff03700 1 --2- 172.21.15.146:0/3376734931 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82780f5410 0x7f82780f5830 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:38.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:38.824+0000 7f827ff03700 1 -- 172.21.15.146:0/3376734931 >> 172.21.15.146:0/3376734931 conn(0x7f82780f0b70 msgr2=0x7f82780f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:38.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:38.824+0000 7f827ff03700 1 -- 172.21.15.146:0/3376734931 shutdown_connections 2024-01-21T20:00:38.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:38.824+0000 7f827ff03700 1 -- 172.21.15.146:0/3376734931 wait complete. 2024-01-21T20:00:38.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:38.825+0000 7f827ff03700 1 Processor -- start 2024-01-21T20:00:38.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:38.825+0000 7f827ff03700 1 -- start start 2024-01-21T20:00:38.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:38.825+0000 7f827ff03700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82780f5410 0x7f8278109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:38.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:38.825+0000 7f827ff03700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8278060240 con 0x7f82780f5410 2024-01-21T20:00:38.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:38.825+0000 7f827dc9f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82780f5410 0x7f8278109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:38.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:38.825+0000 7f827dc9f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82780f5410 0x7f8278109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:39316/0 (socket says 172.21.15.146:39316) 2024-01-21T20:00:38.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:38.825+0000 7f827dc9f700 1 -- 172.21.15.146:0/892908965 learned_addr learned my addr 172.21.15.146:0/892908965 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:00:38.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:38.826+0000 7f827dc9f700 1 -- 172.21.15.146:0/892908965 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8268005040 con 0x7f82780f5410 2024-01-21T20:00:38.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:38.826+0000 7f827dc9f700 1 --2- 172.21.15.146:0/892908965 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82780f5410 0x7f8278109430 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f8268014000 tx=0x7f826800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:38.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:38.826+0000 7f826e7fc700 1 -- 172.21.15.146:0/892908965 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82680095f0 con 0x7f82780f5410 2024-01-21T20:00:38.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:38.826+0000 7f827ff03700 1 -- 172.21.15.146:0/892908965 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8278060440 con 0x7f82780f5410 2024-01-21T20:00:38.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:38.826+0000 7f826e7fc700 1 -- 172.21.15.146:0/892908965 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8268007e70 con 0x7f82780f5410 2024-01-21T20:00:38.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:38.826+0000 7f826e7fc700 1 -- 172.21.15.146:0/892908965 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8268014ea0 con 0x7f82780f5410 2024-01-21T20:00:38.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:38.826+0000 7f827ff03700 1 -- 172.21.15.146:0/892908965 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8278109ae0 con 0x7f82780f5410 2024-01-21T20:00:38.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:38.827+0000 7f826e7fc700 1 -- 172.21.15.146:0/892908965 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8268025070 con 0x7f82780f5410 2024-01-21T20:00:38.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:38.827+0000 7f827ff03700 1 -- 172.21.15.146:0/892908965 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8278045ad0 con 0x7f82780f5410 2024-01-21T20:00:38.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:38.827+0000 7f826e7fc700 1 --2- 172.21.15.146:0/892908965 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f826402ffb0 0x7f8264032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:38.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:38.827+0000 7f826e7fc700 1 -- 172.21.15.146:0/892908965 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f82680409f0 con 0x7f82780f5410 2024-01-21T20:00:38.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:38.827+0000 7f827d49e700 1 --2- 172.21.15.146:0/892908965 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f826402ffb0 0x7f8264032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:38.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:38.828+0000 7f827d49e700 1 --2- 172.21.15.146:0/892908965 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f826402ffb0 0x7f8264032470 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f82740096a0 tx=0x7f8274006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:38.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:38.832+0000 7f826e7fc700 1 -- 172.21.15.146:0/892908965 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f82680079c0 con 0x7f82780f5410 2024-01-21T20:00:38.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:38 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:00:37.137710+0000 mgr.smithi053.fgsmol (mgr.14164) 45 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:39.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:39.033+0000 7f827ff03700 1 -- 172.21.15.146:0/892908965 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f82780fae60 con 0x7f82780f5410 2024-01-21T20:00:39.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:39.034+0000 7f826e7fc700 1 -- 172.21.15.146:0/892908965 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f826800f300 con 0x7f82780f5410 2024-01-21T20:00:39.026 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:00:39.026 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:00:39.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:39.035+0000 7f827ff03700 1 -- 172.21.15.146:0/892908965 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f826402ffb0 msgr2=0x7f8264032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:39.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:39.035+0000 7f827ff03700 1 --2- 172.21.15.146:0/892908965 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f826402ffb0 0x7f8264032470 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f82740096a0 tx=0x7f8274006b40).stop 2024-01-21T20:00:39.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:39.036+0000 7f827ff03700 1 -- 172.21.15.146:0/892908965 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82780f5410 msgr2=0x7f8278109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:39.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:39.036+0000 7f827ff03700 1 --2- 172.21.15.146:0/892908965 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82780f5410 0x7f8278109430 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f8268014000 tx=0x7f826800b040).stop 2024-01-21T20:00:39.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:39.036+0000 7f827ff03700 1 -- 172.21.15.146:0/892908965 shutdown_connections 2024-01-21T20:00:39.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:39.036+0000 7f827ff03700 1 --2- 172.21.15.146:0/892908965 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f826402ffb0 0x7f8264032470 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:39.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:39.036+0000 7f827ff03700 1 --2- 172.21.15.146:0/892908965 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82780f5410 0x7f8278109430 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:39.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:39.036+0000 7f827ff03700 1 -- 172.21.15.146:0/892908965 >> 172.21.15.146:0/892908965 conn(0x7f82780f0b70 msgr2=0x7f82780f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:39.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:39.036+0000 7f827ff03700 1 -- 172.21.15.146:0/892908965 shutdown_connections 2024-01-21T20:00:39.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:39.036+0000 7f827ff03700 1 -- 172.21.15.146:0/892908965 wait complete. 2024-01-21T20:00:39.028 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:00:39.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:39 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:00:39.035254+0000 mon.smithi053 (mon.0) 140 : audit [DBG] from='client.? 172.21.15.146:0/892908965' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:00:40.458 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:00:40.458 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:00:40.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:40 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:00:39.138009+0000 mgr.smithi053.fgsmol (mgr.14164) 46 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:41.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:41.757+0000 7f1c72181700 1 -- 172.21.15.146:0/2123389424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c6c060220 msgr2=0x7f1c6c0f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:41.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:41.757+0000 7f1c72181700 1 --2- 172.21.15.146:0/2123389424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c6c060220 0x7f1c6c0f9150 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f1c5c009fb0 tx=0x7f1c5c009b30).stop 2024-01-21T20:00:41.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:41.758+0000 7f1c72181700 1 -- 172.21.15.146:0/2123389424 shutdown_connections 2024-01-21T20:00:41.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:41.758+0000 7f1c72181700 1 --2- 172.21.15.146:0/2123389424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c6c060220 0x7f1c6c0f9150 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:41.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:41.758+0000 7f1c72181700 1 -- 172.21.15.146:0/2123389424 >> 172.21.15.146:0/2123389424 conn(0x7f1c6c0f0b70 msgr2=0x7f1c6c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:41.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:41.758+0000 7f1c72181700 1 -- 172.21.15.146:0/2123389424 shutdown_connections 2024-01-21T20:00:41.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:41.758+0000 7f1c72181700 1 -- 172.21.15.146:0/2123389424 wait complete. 2024-01-21T20:00:41.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:41.758+0000 7f1c72181700 1 Processor -- start 2024-01-21T20:00:41.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:41.758+0000 7f1c72181700 1 -- start start 2024-01-21T20:00:41.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:41.758+0000 7f1c72181700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c6c060220 0x7f1c6c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:41.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:41.758+0000 7f1c72181700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6c106bd0 con 0x7f1c6c060220 2024-01-21T20:00:41.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:41.759+0000 7f1c6b7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c6c060220 0x7f1c6c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:41.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:41.759+0000 7f1c6b7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c6c060220 0x7f1c6c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:39328/0 (socket says 172.21.15.146:39328) 2024-01-21T20:00:41.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:41.759+0000 7f1c6b7fe700 1 -- 172.21.15.146:0/2767186252 learned_addr learned my addr 172.21.15.146:0/2767186252 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:00:41.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:41.759+0000 7f1c6b7fe700 1 -- 172.21.15.146:0/2767186252 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c5c005040 con 0x7f1c6c060220 2024-01-21T20:00:41.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:41.759+0000 7f1c6b7fe700 1 --2- 172.21.15.146:0/2767186252 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c6c060220 0x7f1c6c1066e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f1c5c013510 tx=0x7f1c5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:41.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:41.759+0000 7f1c63fff700 1 -- 172.21.15.146:0/2767186252 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c5c009320 con 0x7f1c6c060220 2024-01-21T20:00:41.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:41.759+0000 7f1c72181700 1 -- 172.21.15.146:0/2767186252 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c6c106dd0 con 0x7f1c6c060220 2024-01-21T20:00:41.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:41.759+0000 7f1c63fff700 1 -- 172.21.15.146:0/2767186252 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1c5c013f00 con 0x7f1c6c060220 2024-01-21T20:00:41.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:41.759+0000 7f1c63fff700 1 -- 172.21.15.146:0/2767186252 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c5c014dc0 con 0x7f1c6c060220 2024-01-21T20:00:41.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:41.759+0000 7f1c72181700 1 -- 172.21.15.146:0/2767186252 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c6c107260 con 0x7f1c6c060220 2024-01-21T20:00:41.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:41.760+0000 7f1c63fff700 1 -- 172.21.15.146:0/2767186252 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1c5c0095f0 con 0x7f1c6c060220 2024-01-21T20:00:41.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:41.760+0000 7f1c72181700 1 -- 172.21.15.146:0/2767186252 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c6c045ad0 con 0x7f1c6c060220 2024-01-21T20:00:41.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:41.760+0000 7f1c63fff700 1 --2- 172.21.15.146:0/2767186252 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f1c5802ff60 0x7f1c58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:41.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:41.760+0000 7f1c63fff700 1 -- 172.21.15.146:0/2767186252 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1c5c0410c0 con 0x7f1c6c060220 2024-01-21T20:00:41.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:41.761+0000 7f1c6affd700 1 --2- 172.21.15.146:0/2767186252 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f1c5802ff60 0x7f1c58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:41.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:41.761+0000 7f1c6affd700 1 --2- 172.21.15.146:0/2767186252 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f1c5802ff60 0x7f1c58032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f1c540096a0 tx=0x7f1c54006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:41.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:41.765+0000 7f1c63fff700 1 -- 172.21.15.146:0/2767186252 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1c5c0268c0 con 0x7f1c6c060220 2024-01-21T20:00:41.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:41.971+0000 7f1c72181700 1 -- 172.21.15.146:0/2767186252 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1c6c108110 con 0x7f1c6c060220 2024-01-21T20:00:41.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:41.972+0000 7f1c63fff700 1 -- 172.21.15.146:0/2767186252 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1c5c010230 con 0x7f1c6c060220 2024-01-21T20:00:41.963 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:00:41.963 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:00:41.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:41.973+0000 7f1c72181700 1 -- 172.21.15.146:0/2767186252 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f1c5802ff60 msgr2=0x7f1c58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:41.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:41.973+0000 7f1c72181700 1 --2- 172.21.15.146:0/2767186252 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f1c5802ff60 0x7f1c58032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f1c540096a0 tx=0x7f1c54006b40).stop 2024-01-21T20:00:41.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:41.973+0000 7f1c72181700 1 -- 172.21.15.146:0/2767186252 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c6c060220 msgr2=0x7f1c6c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:41.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:41.973+0000 7f1c72181700 1 --2- 172.21.15.146:0/2767186252 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c6c060220 0x7f1c6c1066e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f1c5c013510 tx=0x7f1c5c00b040).stop 2024-01-21T20:00:41.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:41.973+0000 7f1c72181700 1 -- 172.21.15.146:0/2767186252 shutdown_connections 2024-01-21T20:00:41.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:41.973+0000 7f1c72181700 1 --2- 172.21.15.146:0/2767186252 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f1c5802ff60 0x7f1c58032420 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:41.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:41.973+0000 7f1c72181700 1 --2- 172.21.15.146:0/2767186252 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c6c060220 0x7f1c6c1066e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:41.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:41.973+0000 7f1c72181700 1 -- 172.21.15.146:0/2767186252 >> 172.21.15.146:0/2767186252 conn(0x7f1c6c0f0b70 msgr2=0x7f1c6c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:41.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:41.974+0000 7f1c72181700 1 -- 172.21.15.146:0/2767186252 shutdown_connections 2024-01-21T20:00:41.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:41.974+0000 7f1c72181700 1 -- 172.21.15.146:0/2767186252 wait complete. 2024-01-21T20:00:41.966 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:00:42.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:42 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:00:41.138385+0000 mgr.smithi053.fgsmol (mgr.14164) 47 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:42.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:42 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:00:41.973208+0000 mon.smithi053 (mon.0) 141 : audit [DBG] from='client.? 172.21.15.146:0/2767186252' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:00:43.411 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:00:43.412 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:00:44.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:44.623+0000 7f0e81f9b700 1 -- 172.21.15.146:0/3623244611 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e7c0f5410 msgr2=0x7f0e7c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:44.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:44.623+0000 7f0e81f9b700 1 --2- 172.21.15.146:0/3623244611 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e7c0f5410 0x7f0e7c0f5830 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f0e70009fb0 tx=0x7f0e70009b30).stop 2024-01-21T20:00:44.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:44.623+0000 7f0e81f9b700 1 -- 172.21.15.146:0/3623244611 shutdown_connections 2024-01-21T20:00:44.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:44.623+0000 7f0e81f9b700 1 --2- 172.21.15.146:0/3623244611 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e7c0f5410 0x7f0e7c0f5830 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:44.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:44.623+0000 7f0e81f9b700 1 -- 172.21.15.146:0/3623244611 >> 172.21.15.146:0/3623244611 conn(0x7f0e7c0f0b70 msgr2=0x7f0e7c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:44.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:44.623+0000 7f0e81f9b700 1 -- 172.21.15.146:0/3623244611 shutdown_connections 2024-01-21T20:00:44.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:44.623+0000 7f0e81f9b700 1 -- 172.21.15.146:0/3623244611 wait complete. 2024-01-21T20:00:44.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:44.624+0000 7f0e81f9b700 1 Processor -- start 2024-01-21T20:00:44.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:44.624+0000 7f0e81f9b700 1 -- start start 2024-01-21T20:00:44.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:44.624+0000 7f0e81f9b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e7c0f5410 0x7f0e7c109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:44.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:44.624+0000 7f0e81f9b700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e7c060220 con 0x7f0e7c0f5410 2024-01-21T20:00:44.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:44.625+0000 7f0e7b7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e7c0f5410 0x7f0e7c109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:44.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:44.625+0000 7f0e7b7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e7c0f5410 0x7f0e7c109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:39350/0 (socket says 172.21.15.146:39350) 2024-01-21T20:00:44.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:44.625+0000 7f0e7b7fe700 1 -- 172.21.15.146:0/744583942 learned_addr learned my addr 172.21.15.146:0/744583942 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:00:44.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:44.625+0000 7f0e7b7fe700 1 -- 172.21.15.146:0/744583942 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e70005040 con 0x7f0e7c0f5410 2024-01-21T20:00:44.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:44.625+0000 7f0e7b7fe700 1 --2- 172.21.15.146:0/744583942 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e7c0f5410 0x7f0e7c109450 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f0e70013510 tx=0x7f0e7000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:44.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:44.625+0000 7f0e63fff700 1 -- 172.21.15.146:0/744583942 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e70009320 con 0x7f0e7c0f5410 2024-01-21T20:00:44.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:44.625+0000 7f0e81f9b700 1 -- 172.21.15.146:0/744583942 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e7c060420 con 0x7f0e7c0f5410 2024-01-21T20:00:44.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:44.625+0000 7f0e63fff700 1 -- 172.21.15.146:0/744583942 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0e70013f00 con 0x7f0e7c0f5410 2024-01-21T20:00:44.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:44.625+0000 7f0e63fff700 1 -- 172.21.15.146:0/744583942 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e70014d90 con 0x7f0e7c0f5410 2024-01-21T20:00:44.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:44.625+0000 7f0e81f9b700 1 -- 172.21.15.146:0/744583942 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e7c109b00 con 0x7f0e7c0f5410 2024-01-21T20:00:44.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:44.626+0000 7f0e63fff700 1 -- 172.21.15.146:0/744583942 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0e700263a0 con 0x7f0e7c0f5410 2024-01-21T20:00:44.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:44.626+0000 7f0e81f9b700 1 -- 172.21.15.146:0/744583942 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e7c059100 con 0x7f0e7c0f5410 2024-01-21T20:00:44.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:44.626+0000 7f0e63fff700 1 --2- 172.21.15.146:0/744583942 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0e6402ff60 0x7f0e64032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:44.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:44.626+0000 7f0e63fff700 1 -- 172.21.15.146:0/744583942 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0e70041010 con 0x7f0e7c0f5410 2024-01-21T20:00:44.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:44.626+0000 7f0e7affd700 1 --2- 172.21.15.146:0/744583942 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0e6402ff60 0x7f0e64032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:44.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:44.627+0000 7f0e7affd700 1 --2- 172.21.15.146:0/744583942 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0e6402ff60 0x7f0e64032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f0e6c0096a0 tx=0x7f0e6c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:44.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:44.631+0000 7f0e63fff700 1 -- 172.21.15.146:0/744583942 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0e700126f0 con 0x7f0e7c0f5410 2024-01-21T20:00:44.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:44.837+0000 7f0e81f9b700 1 -- 172.21.15.146:0/744583942 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0e7c045ad0 con 0x7f0e7c0f5410 2024-01-21T20:00:44.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:44.837+0000 7f0e63fff700 1 -- 172.21.15.146:0/744583942 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0e70012330 con 0x7f0e7c0f5410 2024-01-21T20:00:44.829 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:00:44.829 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:00:44.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:44.839+0000 7f0e81f9b700 1 -- 172.21.15.146:0/744583942 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0e6402ff60 msgr2=0x7f0e64032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:44.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:44.839+0000 7f0e81f9b700 1 --2- 172.21.15.146:0/744583942 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0e6402ff60 0x7f0e64032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f0e6c0096a0 tx=0x7f0e6c006b40).stop 2024-01-21T20:00:44.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:44.839+0000 7f0e81f9b700 1 -- 172.21.15.146:0/744583942 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e7c0f5410 msgr2=0x7f0e7c109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:44.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:44.839+0000 7f0e81f9b700 1 --2- 172.21.15.146:0/744583942 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e7c0f5410 0x7f0e7c109450 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f0e70013510 tx=0x7f0e7000b040).stop 2024-01-21T20:00:44.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:44.839+0000 7f0e81f9b700 1 -- 172.21.15.146:0/744583942 shutdown_connections 2024-01-21T20:00:44.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:44.839+0000 7f0e81f9b700 1 --2- 172.21.15.146:0/744583942 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0e6402ff60 0x7f0e64032420 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:44.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:44.839+0000 7f0e81f9b700 1 --2- 172.21.15.146:0/744583942 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e7c0f5410 0x7f0e7c109450 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:44.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:44.839+0000 7f0e81f9b700 1 -- 172.21.15.146:0/744583942 >> 172.21.15.146:0/744583942 conn(0x7f0e7c0f0b70 msgr2=0x7f0e7c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:44.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:44.839+0000 7f0e81f9b700 1 -- 172.21.15.146:0/744583942 shutdown_connections 2024-01-21T20:00:44.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:44.839+0000 7f0e81f9b700 1 -- 172.21.15.146:0/744583942 wait complete. 2024-01-21T20:00:44.831 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:00:44.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:44 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:00:43.138777+0000 mgr.smithi053.fgsmol (mgr.14164) 48 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:45.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:45 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:00:44.838744+0000 mon.smithi053 (mon.0) 142 : audit [DBG] from='client.? 172.21.15.146:0/744583942' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:00:46.273 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:00:46.273 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:00:46.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:46 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:00:45.139130+0000 mgr.smithi053.fgsmol (mgr.14164) 49 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:47.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:47.481+0000 7f40c38c8700 1 -- 172.21.15.146:0/3291402663 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40bc0f6510 msgr2=0x7f40bc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:47.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:47.481+0000 7f40c38c8700 1 --2- 172.21.15.146:0/3291402663 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40bc0f6510 0x7f40bc0f6930 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f40ac009fb0 tx=0x7f40ac009b30).stop 2024-01-21T20:00:47.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:47.481+0000 7f40c38c8700 1 -- 172.21.15.146:0/3291402663 shutdown_connections 2024-01-21T20:00:47.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:47.481+0000 7f40c38c8700 1 --2- 172.21.15.146:0/3291402663 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40bc0f6510 0x7f40bc0f6930 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:47.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:47.481+0000 7f40c38c8700 1 -- 172.21.15.146:0/3291402663 >> 172.21.15.146:0/3291402663 conn(0x7f40bc0f1c70 msgr2=0x7f40bc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:47.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:47.481+0000 7f40c38c8700 1 -- 172.21.15.146:0/3291402663 shutdown_connections 2024-01-21T20:00:47.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:47.481+0000 7f40c38c8700 1 -- 172.21.15.146:0/3291402663 wait complete. 2024-01-21T20:00:47.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:47.481+0000 7f40c38c8700 1 Processor -- start 2024-01-21T20:00:47.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:47.481+0000 7f40c38c8700 1 -- start start 2024-01-21T20:00:47.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:47.481+0000 7f40c38c8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40bc0f6510 0x7f40bc06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:47.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:47.482+0000 7f40c38c8700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40bc069e20 con 0x7f40bc0f6510 2024-01-21T20:00:47.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:47.482+0000 7f40c1664700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40bc0f6510 0x7f40bc06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:47.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:47.482+0000 7f40c1664700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40bc0f6510 0x7f40bc06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:42664/0 (socket says 172.21.15.146:42664) 2024-01-21T20:00:47.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:47.482+0000 7f40c1664700 1 -- 172.21.15.146:0/3402387533 learned_addr learned my addr 172.21.15.146:0/3402387533 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:00:47.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:47.482+0000 7f40c1664700 1 -- 172.21.15.146:0/3402387533 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40ac005040 con 0x7f40bc0f6510 2024-01-21T20:00:47.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:47.482+0000 7f40c1664700 1 --2- 172.21.15.146:0/3402387533 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40bc0f6510 0x7f40bc06b9a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f40ac013510 tx=0x7f40ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:47.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:47.482+0000 7f40b1ffb700 1 -- 172.21.15.146:0/3402387533 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40ac009320 con 0x7f40bc0f6510 2024-01-21T20:00:47.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:47.482+0000 7f40b1ffb700 1 -- 172.21.15.146:0/3402387533 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f40ac013f00 con 0x7f40bc0f6510 2024-01-21T20:00:47.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:47.482+0000 7f40c38c8700 1 -- 172.21.15.146:0/3402387533 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40bc069fc0 con 0x7f40bc0f6510 2024-01-21T20:00:47.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:47.482+0000 7f40b1ffb700 1 -- 172.21.15.146:0/3402387533 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40ac014d90 con 0x7f40bc0f6510 2024-01-21T20:00:47.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:47.483+0000 7f40c38c8700 1 -- 172.21.15.146:0/3402387533 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40bc06a450 con 0x7f40bc0f6510 2024-01-21T20:00:47.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:47.483+0000 7f40b1ffb700 1 -- 172.21.15.146:0/3402387533 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f40ac0095f0 con 0x7f40bc0f6510 2024-01-21T20:00:47.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:47.484+0000 7f40b1ffb700 1 --2- 172.21.15.146:0/3402387533 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f40a802ff60 0x7f40a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:47.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:47.484+0000 7f40c38c8700 1 -- 172.21.15.146:0/3402387533 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40bc059100 con 0x7f40bc0f6510 2024-01-21T20:00:47.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:47.484+0000 7f40b1ffb700 1 -- 172.21.15.146:0/3402387533 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f40ac0410c0 con 0x7f40bc0f6510 2024-01-21T20:00:47.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:47.484+0000 7f40c0e63700 1 --2- 172.21.15.146:0/3402387533 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f40a802ff60 0x7f40a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:47.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:47.484+0000 7f40c0e63700 1 --2- 172.21.15.146:0/3402387533 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f40a802ff60 0x7f40a8032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f40b8011eb0 tx=0x7f40b8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:47.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:47.489+0000 7f40b1ffb700 1 -- 172.21.15.146:0/3402387533 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f40ac014420 con 0x7f40bc0f6510 2024-01-21T20:00:47.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:47.693+0000 7f40c38c8700 1 -- 172.21.15.146:0/3402387533 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f40bc0fbf60 con 0x7f40bc0f6510 2024-01-21T20:00:47.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:47.694+0000 7f40b1ffb700 1 -- 172.21.15.146:0/3402387533 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f40ac02b020 con 0x7f40bc0f6510 2024-01-21T20:00:47.686 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:00:47.686 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:00:47.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:47.696+0000 7f40c38c8700 1 -- 172.21.15.146:0/3402387533 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f40a802ff60 msgr2=0x7f40a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:47.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:47.696+0000 7f40c38c8700 1 --2- 172.21.15.146:0/3402387533 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f40a802ff60 0x7f40a8032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f40b8011eb0 tx=0x7f40b8006b40).stop 2024-01-21T20:00:47.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:47.696+0000 7f40c38c8700 1 -- 172.21.15.146:0/3402387533 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40bc0f6510 msgr2=0x7f40bc06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:47.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:47.696+0000 7f40c38c8700 1 --2- 172.21.15.146:0/3402387533 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40bc0f6510 0x7f40bc06b9a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f40ac013510 tx=0x7f40ac00b040).stop 2024-01-21T20:00:47.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:47.696+0000 7f40c38c8700 1 -- 172.21.15.146:0/3402387533 shutdown_connections 2024-01-21T20:00:47.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:47.696+0000 7f40c38c8700 1 --2- 172.21.15.146:0/3402387533 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f40a802ff60 0x7f40a8032420 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:47.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:47.696+0000 7f40c38c8700 1 --2- 172.21.15.146:0/3402387533 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40bc0f6510 0x7f40bc06b9a0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:47.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:47.696+0000 7f40c38c8700 1 -- 172.21.15.146:0/3402387533 >> 172.21.15.146:0/3402387533 conn(0x7f40bc0f1c70 msgr2=0x7f40bc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:47.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:47.696+0000 7f40c38c8700 1 -- 172.21.15.146:0/3402387533 shutdown_connections 2024-01-21T20:00:47.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:47.696+0000 7f40c38c8700 1 -- 172.21.15.146:0/3402387533 wait complete. 2024-01-21T20:00:47.689 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:00:48.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:48 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:00:47.139406+0000 mgr.smithi053.fgsmol (mgr.14164) 50 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:48.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:48 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:00:47.695337+0000 mon.smithi053 (mon.0) 143 : audit [DBG] from='client.? 172.21.15.146:0/3402387533' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:00:49.104 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:00:49.104 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:00:50.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:50.308+0000 7ff4db303700 1 -- 172.21.15.146:0/629166779 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff4d40f5410 msgr2=0x7ff4d40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:50.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:50.308+0000 7ff4db303700 1 --2- 172.21.15.146:0/629166779 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff4d40f5410 0x7ff4d40f5830 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7ff4c4009fb0 tx=0x7ff4c4009b30).stop 2024-01-21T20:00:50.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:50.308+0000 7ff4db303700 1 -- 172.21.15.146:0/629166779 shutdown_connections 2024-01-21T20:00:50.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:50.308+0000 7ff4db303700 1 --2- 172.21.15.146:0/629166779 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff4d40f5410 0x7ff4d40f5830 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:50.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:50.308+0000 7ff4db303700 1 -- 172.21.15.146:0/629166779 >> 172.21.15.146:0/629166779 conn(0x7ff4d40f0b70 msgr2=0x7ff4d40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:50.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:50.308+0000 7ff4db303700 1 -- 172.21.15.146:0/629166779 shutdown_connections 2024-01-21T20:00:50.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:50.308+0000 7ff4db303700 1 -- 172.21.15.146:0/629166779 wait complete. 2024-01-21T20:00:50.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:50.309+0000 7ff4db303700 1 Processor -- start 2024-01-21T20:00:50.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:50.309+0000 7ff4db303700 1 -- start start 2024-01-21T20:00:50.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:50.309+0000 7ff4db303700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff4d40f5410 0x7ff4d4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:50.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:50.309+0000 7ff4db303700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d4105b20 con 0x7ff4d40f5410 2024-01-21T20:00:50.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:50.309+0000 7ff4d909f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff4d40f5410 0x7ff4d4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:50.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:50.309+0000 7ff4d909f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff4d40f5410 0x7ff4d4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:42686/0 (socket says 172.21.15.146:42686) 2024-01-21T20:00:50.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:50.309+0000 7ff4d909f700 1 -- 172.21.15.146:0/2062139147 learned_addr learned my addr 172.21.15.146:0/2062139147 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:00:50.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:50.309+0000 7ff4d909f700 1 -- 172.21.15.146:0/2062139147 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c4005040 con 0x7ff4d40f5410 2024-01-21T20:00:50.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:50.309+0000 7ff4d909f700 1 --2- 172.21.15.146:0/2062139147 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff4d40f5410 0x7ff4d4105630 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7ff4c4013510 tx=0x7ff4c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:50.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:50.310+0000 7ff4c9ffb700 1 -- 172.21.15.146:0/2062139147 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff4c4009320 con 0x7ff4d40f5410 2024-01-21T20:00:50.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:50.310+0000 7ff4db303700 1 -- 172.21.15.146:0/2062139147 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4d4105d20 con 0x7ff4d40f5410 2024-01-21T20:00:50.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:50.310+0000 7ff4c9ffb700 1 -- 172.21.15.146:0/2062139147 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff4c4013f00 con 0x7ff4d40f5410 2024-01-21T20:00:50.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:50.310+0000 7ff4c9ffb700 1 -- 172.21.15.146:0/2062139147 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff4c4014d90 con 0x7ff4d40f5410 2024-01-21T20:00:50.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:50.310+0000 7ff4db303700 1 -- 172.21.15.146:0/2062139147 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4d41061b0 con 0x7ff4d40f5410 2024-01-21T20:00:50.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:50.311+0000 7ff4c9ffb700 1 -- 172.21.15.146:0/2062139147 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff4c40095f0 con 0x7ff4d40f5410 2024-01-21T20:00:50.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:50.311+0000 7ff4db303700 1 -- 172.21.15.146:0/2062139147 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4d4045ad0 con 0x7ff4d40f5410 2024-01-21T20:00:50.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:50.311+0000 7ff4c9ffb700 1 --2- 172.21.15.146:0/2062139147 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff4c0030000 0x7ff4c00324c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:50.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:50.311+0000 7ff4c9ffb700 1 -- 172.21.15.146:0/2062139147 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff4c4041040 con 0x7ff4d40f5410 2024-01-21T20:00:50.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:50.311+0000 7ff4d889e700 1 --2- 172.21.15.146:0/2062139147 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff4c0030000 0x7ff4c00324c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:50.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:50.311+0000 7ff4d889e700 1 --2- 172.21.15.146:0/2062139147 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff4c0030000 0x7ff4c00324c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7ff4d00096a0 tx=0x7ff4d0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:50.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:50.315+0000 7ff4c9ffb700 1 -- 172.21.15.146:0/2062139147 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff4c4014420 con 0x7ff4d40f5410 2024-01-21T20:00:50.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:50.521+0000 7ff4db303700 1 -- 172.21.15.146:0/2062139147 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff4d40fae60 con 0x7ff4d40f5410 2024-01-21T20:00:50.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:50.522+0000 7ff4c9ffb700 1 -- 172.21.15.146:0/2062139147 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff4c401d7f0 con 0x7ff4d40f5410 2024-01-21T20:00:50.513 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:00:50.513 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:00:50.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:50.523+0000 7ff4db303700 1 -- 172.21.15.146:0/2062139147 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff4c0030000 msgr2=0x7ff4c00324c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:50.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:50.523+0000 7ff4db303700 1 --2- 172.21.15.146:0/2062139147 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff4c0030000 0x7ff4c00324c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7ff4d00096a0 tx=0x7ff4d0006b40).stop 2024-01-21T20:00:50.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:50.523+0000 7ff4db303700 1 -- 172.21.15.146:0/2062139147 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff4d40f5410 msgr2=0x7ff4d4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:50.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:50.523+0000 7ff4db303700 1 --2- 172.21.15.146:0/2062139147 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff4d40f5410 0x7ff4d4105630 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7ff4c4013510 tx=0x7ff4c400b040).stop 2024-01-21T20:00:50.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:50.523+0000 7ff4db303700 1 -- 172.21.15.146:0/2062139147 shutdown_connections 2024-01-21T20:00:50.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:50.523+0000 7ff4db303700 1 --2- 172.21.15.146:0/2062139147 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff4c0030000 0x7ff4c00324c0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:50.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:50.523+0000 7ff4db303700 1 --2- 172.21.15.146:0/2062139147 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff4d40f5410 0x7ff4d4105630 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:50.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:50.523+0000 7ff4db303700 1 -- 172.21.15.146:0/2062139147 >> 172.21.15.146:0/2062139147 conn(0x7ff4d40f0b70 msgr2=0x7ff4d40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:50.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:50.523+0000 7ff4db303700 1 -- 172.21.15.146:0/2062139147 shutdown_connections 2024-01-21T20:00:50.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:50.523+0000 7ff4db303700 1 -- 172.21.15.146:0/2062139147 wait complete. 2024-01-21T20:00:50.516 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:00:50.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:50 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:00:49.139694+0000 mgr.smithi053.fgsmol (mgr.14164) 51 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:50.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:50 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:00:50.522987+0000 mon.smithi053 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.146:0/2062139147' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:00:51.921 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:00:51.921 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:00:52.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:52 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:00:51.140024+0000 mgr.smithi053.fgsmol (mgr.14164) 52 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:53.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:53.120+0000 7f76c1fbd700 1 -- 172.21.15.146:0/2184198559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76bc0f5410 msgr2=0x7f76bc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:53.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:53.120+0000 7f76c1fbd700 1 --2- 172.21.15.146:0/2184198559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76bc0f5410 0x7f76bc0f5830 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f76ac009fb0 tx=0x7f76ac009b30).stop 2024-01-21T20:00:53.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:53.120+0000 7f76c1fbd700 1 -- 172.21.15.146:0/2184198559 shutdown_connections 2024-01-21T20:00:53.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:53.120+0000 7f76c1fbd700 1 --2- 172.21.15.146:0/2184198559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76bc0f5410 0x7f76bc0f5830 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:53.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:53.120+0000 7f76c1fbd700 1 -- 172.21.15.146:0/2184198559 >> 172.21.15.146:0/2184198559 conn(0x7f76bc0f0b70 msgr2=0x7f76bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:53.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:53.120+0000 7f76c1fbd700 1 -- 172.21.15.146:0/2184198559 shutdown_connections 2024-01-21T20:00:53.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:53.120+0000 7f76c1fbd700 1 -- 172.21.15.146:0/2184198559 wait complete. 2024-01-21T20:00:53.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:53.121+0000 7f76c1fbd700 1 Processor -- start 2024-01-21T20:00:53.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:53.121+0000 7f76c1fbd700 1 -- start start 2024-01-21T20:00:53.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:53.121+0000 7f76c1fbd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76bc0f5410 0x7f76bc105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:53.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:53.121+0000 7f76c1fbd700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76bc105b20 con 0x7f76bc0f5410 2024-01-21T20:00:53.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:53.122+0000 7f76bb7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76bc0f5410 0x7f76bc105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:53.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:53.122+0000 7f76bb7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76bc0f5410 0x7f76bc105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:42700/0 (socket says 172.21.15.146:42700) 2024-01-21T20:00:53.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:53.122+0000 7f76bb7fe700 1 -- 172.21.15.146:0/2909988675 learned_addr learned my addr 172.21.15.146:0/2909988675 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:00:53.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:53.122+0000 7f76bb7fe700 1 -- 172.21.15.146:0/2909988675 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76ac005040 con 0x7f76bc0f5410 2024-01-21T20:00:53.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:53.122+0000 7f76bb7fe700 1 --2- 172.21.15.146:0/2909988675 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76bc0f5410 0x7f76bc105630 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f76ac014000 tx=0x7f76ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:53.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:53.122+0000 7f76a3fff700 1 -- 172.21.15.146:0/2909988675 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76ac0095f0 con 0x7f76bc0f5410 2024-01-21T20:00:53.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:53.122+0000 7f76c1fbd700 1 -- 172.21.15.146:0/2909988675 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76bc105d20 con 0x7f76bc0f5410 2024-01-21T20:00:53.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:53.122+0000 7f76a3fff700 1 -- 172.21.15.146:0/2909988675 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f76ac007e70 con 0x7f76bc0f5410 2024-01-21T20:00:53.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:53.122+0000 7f76a3fff700 1 -- 172.21.15.146:0/2909988675 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76ac014ea0 con 0x7f76bc0f5410 2024-01-21T20:00:53.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:53.123+0000 7f76c1fbd700 1 -- 172.21.15.146:0/2909988675 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76bc1061b0 con 0x7f76bc0f5410 2024-01-21T20:00:53.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:53.123+0000 7f76a3fff700 1 -- 172.21.15.146:0/2909988675 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f76ac025070 con 0x7f76bc0f5410 2024-01-21T20:00:53.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:53.123+0000 7f76c1fbd700 1 -- 172.21.15.146:0/2909988675 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76bc045ad0 con 0x7f76bc0f5410 2024-01-21T20:00:53.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:53.123+0000 7f76a3fff700 1 --2- 172.21.15.146:0/2909988675 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f76a402ff60 0x7f76a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:53.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:53.123+0000 7f76a3fff700 1 -- 172.21.15.146:0/2909988675 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f76ac040a30 con 0x7f76bc0f5410 2024-01-21T20:00:53.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:53.124+0000 7f76baffd700 1 --2- 172.21.15.146:0/2909988675 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f76a402ff60 0x7f76a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:53.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:53.124+0000 7f76baffd700 1 --2- 172.21.15.146:0/2909988675 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f76a402ff60 0x7f76a4032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f76b00096a0 tx=0x7f76b0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:53.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:53.128+0000 7f76a3fff700 1 -- 172.21.15.146:0/2909988675 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f76ac0079c0 con 0x7f76bc0f5410 2024-01-21T20:00:53.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:53.334+0000 7f76c1fbd700 1 -- 172.21.15.146:0/2909988675 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f76bc0fae60 con 0x7f76bc0f5410 2024-01-21T20:00:53.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:53.335+0000 7f76a3fff700 1 -- 172.21.15.146:0/2909988675 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f76ac00f300 con 0x7f76bc0f5410 2024-01-21T20:00:53.327 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:00:53.327 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:00:53.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:53.336+0000 7f76c1fbd700 1 -- 172.21.15.146:0/2909988675 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f76a402ff60 msgr2=0x7f76a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:53.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:53.336+0000 7f76c1fbd700 1 --2- 172.21.15.146:0/2909988675 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f76a402ff60 0x7f76a4032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f76b00096a0 tx=0x7f76b0006b40).stop 2024-01-21T20:00:53.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:53.336+0000 7f76c1fbd700 1 -- 172.21.15.146:0/2909988675 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76bc0f5410 msgr2=0x7f76bc105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:53.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:53.337+0000 7f76c1fbd700 1 --2- 172.21.15.146:0/2909988675 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76bc0f5410 0x7f76bc105630 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f76ac014000 tx=0x7f76ac00b040).stop 2024-01-21T20:00:53.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:53.337+0000 7f76c1fbd700 1 -- 172.21.15.146:0/2909988675 shutdown_connections 2024-01-21T20:00:53.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:53.337+0000 7f76c1fbd700 1 --2- 172.21.15.146:0/2909988675 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f76a402ff60 0x7f76a4032420 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:53.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:53.337+0000 7f76c1fbd700 1 --2- 172.21.15.146:0/2909988675 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76bc0f5410 0x7f76bc105630 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:53.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:53.337+0000 7f76c1fbd700 1 -- 172.21.15.146:0/2909988675 >> 172.21.15.146:0/2909988675 conn(0x7f76bc0f0b70 msgr2=0x7f76bc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:53.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:53.337+0000 7f76c1fbd700 1 -- 172.21.15.146:0/2909988675 shutdown_connections 2024-01-21T20:00:53.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:53.337+0000 7f76c1fbd700 1 -- 172.21.15.146:0/2909988675 wait complete. 2024-01-21T20:00:53.329 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:00:53.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:53 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:00:53.336326+0000 mon.smithi053 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.146:0/2909988675' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:00:54.757 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:00:54.757 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:00:54.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:54 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:00:53.140376+0000 mgr.smithi053.fgsmol (mgr.14164) 53 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:56.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:55.964+0000 7fc103173700 1 -- 172.21.15.146:0/1229721239 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0fc0f51a0 msgr2=0x7fc0fc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:56.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:55.964+0000 7fc103173700 1 --2- 172.21.15.146:0/1229721239 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0fc0f51a0 0x7fc0fc0f55c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fc0ec009fb0 tx=0x7fc0ec009b30).stop 2024-01-21T20:00:56.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:55.965+0000 7fc103173700 1 -- 172.21.15.146:0/1229721239 shutdown_connections 2024-01-21T20:00:56.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:55.965+0000 7fc103173700 1 --2- 172.21.15.146:0/1229721239 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0fc0f51a0 0x7fc0fc0f55c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:56.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:55.965+0000 7fc103173700 1 -- 172.21.15.146:0/1229721239 >> 172.21.15.146:0/1229721239 conn(0x7fc0fc0f09a0 msgr2=0x7fc0fc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:56.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:55.965+0000 7fc103173700 1 -- 172.21.15.146:0/1229721239 shutdown_connections 2024-01-21T20:00:56.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:55.965+0000 7fc103173700 1 -- 172.21.15.146:0/1229721239 wait complete. 2024-01-21T20:00:56.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:55.965+0000 7fc103173700 1 Processor -- start 2024-01-21T20:00:56.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:55.965+0000 7fc103173700 1 -- start start 2024-01-21T20:00:56.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:55.965+0000 7fc103173700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0fc0f51a0 0x7fc0fc1053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:56.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:55.965+0000 7fc103173700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0fc1058a0 con 0x7fc0fc0f51a0 2024-01-21T20:00:56.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:55.965+0000 7fc102171700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0fc0f51a0 0x7fc0fc1053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:56.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:55.965+0000 7fc102171700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0fc0f51a0 0x7fc0fc1053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:42716/0 (socket says 172.21.15.146:42716) 2024-01-21T20:00:56.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:55.965+0000 7fc102171700 1 -- 172.21.15.146:0/1011547542 learned_addr learned my addr 172.21.15.146:0/1011547542 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:00:56.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:55.966+0000 7fc102171700 1 -- 172.21.15.146:0/1011547542 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0ec005040 con 0x7fc0fc0f51a0 2024-01-21T20:00:56.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:55.966+0000 7fc102171700 1 --2- 172.21.15.146:0/1011547542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0fc0f51a0 0x7fc0fc1053b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fc0ec013510 tx=0x7fc0ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:56.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:55.966+0000 7fc0f2ffd700 1 -- 172.21.15.146:0/1011547542 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0ec009320 con 0x7fc0fc0f51a0 2024-01-21T20:00:56.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:55.966+0000 7fc0f2ffd700 1 -- 172.21.15.146:0/1011547542 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc0ec013f00 con 0x7fc0fc0f51a0 2024-01-21T20:00:56.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:55.966+0000 7fc103173700 1 -- 172.21.15.146:0/1011547542 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0fc105aa0 con 0x7fc0fc0f51a0 2024-01-21T20:00:56.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:55.966+0000 7fc0f2ffd700 1 -- 172.21.15.146:0/1011547542 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0ec014dc0 con 0x7fc0fc0f51a0 2024-01-21T20:00:56.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:55.966+0000 7fc103173700 1 -- 172.21.15.146:0/1011547542 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0fc105f30 con 0x7fc0fc0f51a0 2024-01-21T20:00:56.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:55.967+0000 7fc0f2ffd700 1 -- 172.21.15.146:0/1011547542 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc0ec0095f0 con 0x7fc0fc0f51a0 2024-01-21T20:00:56.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:55.967+0000 7fc103173700 1 -- 172.21.15.146:0/1011547542 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0fc045ad0 con 0x7fc0fc0f51a0 2024-01-21T20:00:56.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:55.967+0000 7fc0f2ffd700 1 --2- 172.21.15.146:0/1011547542 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fc0e802ff60 0x7fc0e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:56.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:55.967+0000 7fc0f2ffd700 1 -- 172.21.15.146:0/1011547542 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc0ec0410c0 con 0x7fc0fc0f51a0 2024-01-21T20:00:56.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:55.967+0000 7fc101970700 1 --2- 172.21.15.146:0/1011547542 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fc0e802ff60 0x7fc0e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:56.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:55.968+0000 7fc101970700 1 --2- 172.21.15.146:0/1011547542 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fc0e802ff60 0x7fc0e8032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fc0f80096a0 tx=0x7fc0f8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:56.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:55.972+0000 7fc0f2ffd700 1 -- 172.21.15.146:0/1011547542 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc0ec014420 con 0x7fc0fc0f51a0 2024-01-21T20:00:56.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:56.178+0000 7fc103173700 1 -- 172.21.15.146:0/1011547542 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc0fc106de0 con 0x7fc0fc0f51a0 2024-01-21T20:00:56.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:56.178+0000 7fc0f2ffd700 1 -- 172.21.15.146:0/1011547542 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc0ec010230 con 0x7fc0fc0f51a0 2024-01-21T20:00:56.170 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:00:56.170 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:00:56.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:56.180+0000 7fc103173700 1 -- 172.21.15.146:0/1011547542 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fc0e802ff60 msgr2=0x7fc0e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:56.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:56.180+0000 7fc103173700 1 --2- 172.21.15.146:0/1011547542 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fc0e802ff60 0x7fc0e8032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fc0f80096a0 tx=0x7fc0f8006b40).stop 2024-01-21T20:00:56.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:56.180+0000 7fc103173700 1 -- 172.21.15.146:0/1011547542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0fc0f51a0 msgr2=0x7fc0fc1053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:56.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:56.180+0000 7fc103173700 1 --2- 172.21.15.146:0/1011547542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0fc0f51a0 0x7fc0fc1053b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fc0ec013510 tx=0x7fc0ec00b040).stop 2024-01-21T20:00:56.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:56.180+0000 7fc103173700 1 -- 172.21.15.146:0/1011547542 shutdown_connections 2024-01-21T20:00:56.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:56.180+0000 7fc103173700 1 --2- 172.21.15.146:0/1011547542 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fc0e802ff60 0x7fc0e8032420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:56.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:56.180+0000 7fc103173700 1 --2- 172.21.15.146:0/1011547542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0fc0f51a0 0x7fc0fc1053b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:56.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:56.180+0000 7fc103173700 1 -- 172.21.15.146:0/1011547542 >> 172.21.15.146:0/1011547542 conn(0x7fc0fc0f09a0 msgr2=0x7fc0fc0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:56.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:56.180+0000 7fc103173700 1 -- 172.21.15.146:0/1011547542 shutdown_connections 2024-01-21T20:00:56.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:56.180+0000 7fc103173700 1 -- 172.21.15.146:0/1011547542 wait complete. 2024-01-21T20:00:56.172 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:00:56.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:56 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:00:55.140585+0000 mgr.smithi053.fgsmol (mgr.14164) 54 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:56.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:56 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:00:56.179907+0000 mon.smithi053 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.146:0/1011547542' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:00:57.601 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:00:57.602 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:00:58.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:58.867+0000 7efec2dd3700 1 -- 172.21.15.146:0/1585822701 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efebc0f6510 msgr2=0x7efebc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:58.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:58.867+0000 7efec2dd3700 1 --2- 172.21.15.146:0/1585822701 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efebc0f6510 0x7efebc0f6930 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7efeb0009fb0 tx=0x7efeb0009b30).stop 2024-01-21T20:00:58.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:58.867+0000 7efec2dd3700 1 -- 172.21.15.146:0/1585822701 shutdown_connections 2024-01-21T20:00:58.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:58.867+0000 7efec2dd3700 1 --2- 172.21.15.146:0/1585822701 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efebc0f6510 0x7efebc0f6930 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:58.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:58.867+0000 7efec2dd3700 1 -- 172.21.15.146:0/1585822701 >> 172.21.15.146:0/1585822701 conn(0x7efebc0f1c70 msgr2=0x7efebc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:58.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:58.868+0000 7efec2dd3700 1 -- 172.21.15.146:0/1585822701 shutdown_connections 2024-01-21T20:00:58.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:58.868+0000 7efec2dd3700 1 -- 172.21.15.146:0/1585822701 wait complete. 2024-01-21T20:00:58.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:58.868+0000 7efec2dd3700 1 Processor -- start 2024-01-21T20:00:58.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:58.868+0000 7efec2dd3700 1 -- start start 2024-01-21T20:00:58.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:58.868+0000 7efec2dd3700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efebc0f6510 0x7efebc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:58.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:58.868+0000 7efec2dd3700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efebc106c20 con 0x7efebc0f6510 2024-01-21T20:00:58.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:58.869+0000 7efec0b6f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efebc0f6510 0x7efebc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:58.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:58.869+0000 7efec0b6f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efebc0f6510 0x7efebc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:35058/0 (socket says 172.21.15.146:35058) 2024-01-21T20:00:58.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:58.869+0000 7efec0b6f700 1 -- 172.21.15.146:0/858286744 learned_addr learned my addr 172.21.15.146:0/858286744 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:00:58.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:58.869+0000 7efec0b6f700 1 -- 172.21.15.146:0/858286744 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb0005040 con 0x7efebc0f6510 2024-01-21T20:00:58.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:58.869+0000 7efec0b6f700 1 --2- 172.21.15.146:0/858286744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efebc0f6510 0x7efebc106730 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7efeb0013510 tx=0x7efeb000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:58.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:58.869+0000 7efeb97fa700 1 -- 172.21.15.146:0/858286744 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efeb0009320 con 0x7efebc0f6510 2024-01-21T20:00:58.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:58.869+0000 7efec2dd3700 1 -- 172.21.15.146:0/858286744 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efebc106e20 con 0x7efebc0f6510 2024-01-21T20:00:58.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:58.869+0000 7efeb97fa700 1 -- 172.21.15.146:0/858286744 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efeb0013f00 con 0x7efebc0f6510 2024-01-21T20:00:58.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:58.869+0000 7efeb97fa700 1 -- 172.21.15.146:0/858286744 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efeb0014d90 con 0x7efebc0f6510 2024-01-21T20:00:58.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:58.869+0000 7efec2dd3700 1 -- 172.21.15.146:0/858286744 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efebc1072b0 con 0x7efebc0f6510 2024-01-21T20:00:58.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:58.870+0000 7efeb97fa700 1 -- 172.21.15.146:0/858286744 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efeb0014420 con 0x7efebc0f6510 2024-01-21T20:00:58.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:58.870+0000 7efec2dd3700 1 -- 172.21.15.146:0/858286744 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efebc045ad0 con 0x7efebc0f6510 2024-01-21T20:00:58.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:58.870+0000 7efeb97fa700 1 --2- 172.21.15.146:0/858286744 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7efeac02ff60 0x7efeac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:58.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:58.870+0000 7efeb97fa700 1 -- 172.21.15.146:0/858286744 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efeb00410c0 con 0x7efebc0f6510 2024-01-21T20:00:58.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:58.870+0000 7efebbfff700 1 --2- 172.21.15.146:0/858286744 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7efeac02ff60 0x7efeac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:58.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:58.871+0000 7efebbfff700 1 --2- 172.21.15.146:0/858286744 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7efeac02ff60 0x7efeac032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7efea80096a0 tx=0x7efea8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:58.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:58.875+0000 7efeb97fa700 1 -- 172.21.15.146:0/858286744 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efeb00095f0 con 0x7efebc0f6510 2024-01-21T20:00:58.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:58 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:00:57.140932+0000 mgr.smithi053.fgsmol (mgr.14164) 55 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:59.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:59.077+0000 7efec2dd3700 1 -- 172.21.15.146:0/858286744 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efebc0fbf60 con 0x7efebc0f6510 2024-01-21T20:00:59.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:59.078+0000 7efeb97fa700 1 -- 172.21.15.146:0/858286744 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efeb001db70 con 0x7efebc0f6510 2024-01-21T20:00:59.069 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:00:59.069 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:00:59.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:59.079+0000 7efec2dd3700 1 -- 172.21.15.146:0/858286744 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7efeac02ff60 msgr2=0x7efeac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:59.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:59.079+0000 7efec2dd3700 1 --2- 172.21.15.146:0/858286744 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7efeac02ff60 0x7efeac032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7efea80096a0 tx=0x7efea8006b40).stop 2024-01-21T20:00:59.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:59.079+0000 7efec2dd3700 1 -- 172.21.15.146:0/858286744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efebc0f6510 msgr2=0x7efebc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:59.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:59.079+0000 7efec2dd3700 1 --2- 172.21.15.146:0/858286744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efebc0f6510 0x7efebc106730 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7efeb0013510 tx=0x7efeb000b040).stop 2024-01-21T20:00:59.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:59.079+0000 7efec2dd3700 1 -- 172.21.15.146:0/858286744 shutdown_connections 2024-01-21T20:00:59.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:59.079+0000 7efec2dd3700 1 --2- 172.21.15.146:0/858286744 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7efeac02ff60 0x7efeac032420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:59.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:59.079+0000 7efec2dd3700 1 --2- 172.21.15.146:0/858286744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efebc0f6510 0x7efebc106730 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:59.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:59.079+0000 7efec2dd3700 1 -- 172.21.15.146:0/858286744 >> 172.21.15.146:0/858286744 conn(0x7efebc0f1c70 msgr2=0x7efebc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:59.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:59.079+0000 7efec2dd3700 1 -- 172.21.15.146:0/858286744 shutdown_connections 2024-01-21T20:00:59.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:00:59.079+0000 7efec2dd3700 1 -- 172.21.15.146:0/858286744 wait complete. 2024-01-21T20:00:59.072 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:00:59.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:00:59 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:00:59.078907+0000 mon.smithi053 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.146:0/858286744' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:01:00.510 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:01:00.510 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:01:00.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:00 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:00:59.141330+0000 mgr.smithi053.fgsmol (mgr.14164) 56 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:01.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:01.733+0000 7fa0b913a700 1 -- 172.21.15.146:0/1345405430 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0b40f5330 msgr2=0x7fa0b40f5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:01.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:01.733+0000 7fa0b913a700 1 --2- 172.21.15.146:0/1345405430 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0b40f5330 0x7fa0b40f5750 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fa09c009fb0 tx=0x7fa09c009b30).stop 2024-01-21T20:01:01.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:01.733+0000 7fa0b913a700 1 -- 172.21.15.146:0/1345405430 shutdown_connections 2024-01-21T20:01:01.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:01.733+0000 7fa0b913a700 1 --2- 172.21.15.146:0/1345405430 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0b40f5330 0x7fa0b40f5750 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:01.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:01.733+0000 7fa0b913a700 1 -- 172.21.15.146:0/1345405430 >> 172.21.15.146:0/1345405430 conn(0x7fa0b40f0a90 msgr2=0x7fa0b40f2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:01.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:01.733+0000 7fa0b913a700 1 -- 172.21.15.146:0/1345405430 shutdown_connections 2024-01-21T20:01:01.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:01.733+0000 7fa0b913a700 1 -- 172.21.15.146:0/1345405430 wait complete. 2024-01-21T20:01:01.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:01.734+0000 7fa0b913a700 1 Processor -- start 2024-01-21T20:01:01.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:01.734+0000 7fa0b913a700 1 -- start start 2024-01-21T20:01:01.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:01.734+0000 7fa0b913a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0b40f5330 0x7fa0b41058d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:01.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:01.734+0000 7fa0b913a700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b4105dc0 con 0x7fa0b40f5330 2024-01-21T20:01:01.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:01.735+0000 7fa0b2d9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0b40f5330 0x7fa0b41058d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:01.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:01.735+0000 7fa0b2d9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0b40f5330 0x7fa0b41058d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:35078/0 (socket says 172.21.15.146:35078) 2024-01-21T20:01:01.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:01.735+0000 7fa0b2d9d700 1 -- 172.21.15.146:0/228575328 learned_addr learned my addr 172.21.15.146:0/228575328 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:01:01.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:01.735+0000 7fa0b2d9d700 1 -- 172.21.15.146:0/228575328 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa09c005040 con 0x7fa0b40f5330 2024-01-21T20:01:01.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:01.735+0000 7fa0b2d9d700 1 --2- 172.21.15.146:0/228575328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0b40f5330 0x7fa0b41058d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fa09c013510 tx=0x7fa09c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:01.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:01.735+0000 7fa0ab7fe700 1 -- 172.21.15.146:0/228575328 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa09c009320 con 0x7fa0b40f5330 2024-01-21T20:01:01.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:01.735+0000 7fa0b913a700 1 -- 172.21.15.146:0/228575328 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0b4105fc0 con 0x7fa0b40f5330 2024-01-21T20:01:01.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:01.735+0000 7fa0ab7fe700 1 -- 172.21.15.146:0/228575328 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa09c013f00 con 0x7fa0b40f5330 2024-01-21T20:01:01.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:01.735+0000 7fa0ab7fe700 1 -- 172.21.15.146:0/228575328 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa09c014d90 con 0x7fa0b40f5330 2024-01-21T20:01:01.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:01.735+0000 7fa0b913a700 1 -- 172.21.15.146:0/228575328 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0b4106450 con 0x7fa0b40f5330 2024-01-21T20:01:01.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:01.736+0000 7fa0ab7fe700 1 -- 172.21.15.146:0/228575328 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa09c0095f0 con 0x7fa0b40f5330 2024-01-21T20:01:01.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:01.736+0000 7fa0b913a700 1 -- 172.21.15.146:0/228575328 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0b4045ad0 con 0x7fa0b40f5330 2024-01-21T20:01:01.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:01.736+0000 7fa0ab7fe700 1 --2- 172.21.15.146:0/228575328 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa0a002ff60 0x7fa0a0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:01.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:01.736+0000 7fa0ab7fe700 1 -- 172.21.15.146:0/228575328 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa09c041000 con 0x7fa0b40f5330 2024-01-21T20:01:01.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:01.737+0000 7fa0b259c700 1 --2- 172.21.15.146:0/228575328 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa0a002ff60 0x7fa0a0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:01.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:01.737+0000 7fa0b259c700 1 --2- 172.21.15.146:0/228575328 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa0a002ff60 0x7fa0a0032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fa0a40096a0 tx=0x7fa0a4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:01.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:01.741+0000 7fa0ab7fe700 1 -- 172.21.15.146:0/228575328 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa09c014420 con 0x7fa0b40f5330 2024-01-21T20:01:01.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:01.947+0000 7fa0b913a700 1 -- 172.21.15.146:0/228575328 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa0b405ea50 con 0x7fa0b40f5330 2024-01-21T20:01:01.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:01.948+0000 7fa0ab7fe700 1 -- 172.21.15.146:0/228575328 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa09c010230 con 0x7fa0b40f5330 2024-01-21T20:01:01.939 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:01:01.939 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:01:01.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:01.949+0000 7fa0b913a700 1 -- 172.21.15.146:0/228575328 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa0a002ff60 msgr2=0x7fa0a0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:01.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:01.949+0000 7fa0b913a700 1 --2- 172.21.15.146:0/228575328 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa0a002ff60 0x7fa0a0032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fa0a40096a0 tx=0x7fa0a4006b40).stop 2024-01-21T20:01:01.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:01.949+0000 7fa0b913a700 1 -- 172.21.15.146:0/228575328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0b40f5330 msgr2=0x7fa0b41058d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:01.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:01.949+0000 7fa0b913a700 1 --2- 172.21.15.146:0/228575328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0b40f5330 0x7fa0b41058d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fa09c013510 tx=0x7fa09c00b040).stop 2024-01-21T20:01:01.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:01.949+0000 7fa0b913a700 1 -- 172.21.15.146:0/228575328 shutdown_connections 2024-01-21T20:01:01.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:01.949+0000 7fa0b913a700 1 --2- 172.21.15.146:0/228575328 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa0a002ff60 0x7fa0a0032420 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:01.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:01.949+0000 7fa0b913a700 1 --2- 172.21.15.146:0/228575328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0b40f5330 0x7fa0b41058d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:01.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:01.949+0000 7fa0b913a700 1 -- 172.21.15.146:0/228575328 >> 172.21.15.146:0/228575328 conn(0x7fa0b40f0a90 msgr2=0x7fa0b40f1530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:01.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:01.949+0000 7fa0b913a700 1 -- 172.21.15.146:0/228575328 shutdown_connections 2024-01-21T20:01:01.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:01.949+0000 7fa0b913a700 1 -- 172.21.15.146:0/228575328 wait complete. 2024-01-21T20:01:01.942 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:01:02.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:02 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:01:01.141622+0000 mgr.smithi053.fgsmol (mgr.14164) 57 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:02.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:02 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:01:01.948707+0000 mon.smithi053 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.146:0/228575328' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:01:03.339 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:01:03.339 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:01:04.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:04.553+0000 7f0b0f2f0700 1 -- 172.21.15.146:0/3100892387 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0b080601c0 msgr2=0x7f0b080f91b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:04.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:04.553+0000 7f0b0f2f0700 1 --2- 172.21.15.146:0/3100892387 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0b080601c0 0x7f0b080f91b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f0b04009fb0 tx=0x7f0b04009b30).stop 2024-01-21T20:01:04.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:04.553+0000 7f0b0f2f0700 1 -- 172.21.15.146:0/3100892387 shutdown_connections 2024-01-21T20:01:04.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:04.553+0000 7f0b0f2f0700 1 --2- 172.21.15.146:0/3100892387 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0b080601c0 0x7f0b080f91b0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:04.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:04.553+0000 7f0b0f2f0700 1 -- 172.21.15.146:0/3100892387 >> 172.21.15.146:0/3100892387 conn(0x7f0b080f0b70 msgr2=0x7f0b080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:04.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:04.553+0000 7f0b0f2f0700 1 -- 172.21.15.146:0/3100892387 shutdown_connections 2024-01-21T20:01:04.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:04.553+0000 7f0b0f2f0700 1 -- 172.21.15.146:0/3100892387 wait complete. 2024-01-21T20:01:04.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:04.554+0000 7f0b0f2f0700 1 Processor -- start 2024-01-21T20:01:04.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:04.554+0000 7f0b0f2f0700 1 -- start start 2024-01-21T20:01:04.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:04.554+0000 7f0b0f2f0700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0b080601c0 0x7f0b080f6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:04.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:04.554+0000 7f0b0f2f0700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b080f50b0 con 0x7f0b080601c0 2024-01-21T20:01:04.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:04.554+0000 7f0b0d08c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0b080601c0 0x7f0b080f6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:04.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:04.554+0000 7f0b0d08c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0b080601c0 0x7f0b080f6c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:35096/0 (socket says 172.21.15.146:35096) 2024-01-21T20:01:04.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:04.555+0000 7f0b0d08c700 1 -- 172.21.15.146:0/4041614495 learned_addr learned my addr 172.21.15.146:0/4041614495 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:01:04.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:04.555+0000 7f0b0d08c700 1 -- 172.21.15.146:0/4041614495 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b04005040 con 0x7f0b080601c0 2024-01-21T20:01:04.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:04.555+0000 7f0b0d08c700 1 --2- 172.21.15.146:0/4041614495 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0b080601c0 0x7f0b080f6c30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f0b04013510 tx=0x7f0b0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:04.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:04.555+0000 7f0afdffb700 1 -- 172.21.15.146:0/4041614495 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b04009320 con 0x7f0b080601c0 2024-01-21T20:01:04.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:04.555+0000 7f0b0f2f0700 1 -- 172.21.15.146:0/4041614495 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b080f5250 con 0x7f0b080601c0 2024-01-21T20:01:04.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:04.555+0000 7f0afdffb700 1 -- 172.21.15.146:0/4041614495 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0b04013f00 con 0x7f0b080601c0 2024-01-21T20:01:04.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:04.555+0000 7f0afdffb700 1 -- 172.21.15.146:0/4041614495 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b04014dc0 con 0x7f0b080601c0 2024-01-21T20:01:04.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:04.555+0000 7f0b0f2f0700 1 -- 172.21.15.146:0/4041614495 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b080f56e0 con 0x7f0b080601c0 2024-01-21T20:01:04.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:04.556+0000 7f0afdffb700 1 -- 172.21.15.146:0/4041614495 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0b0401d3d0 con 0x7f0b080601c0 2024-01-21T20:01:04.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:04.556+0000 7f0b0f2f0700 1 -- 172.21.15.146:0/4041614495 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b08059100 con 0x7f0b080601c0 2024-01-21T20:01:04.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:04.556+0000 7f0afdffb700 1 --2- 172.21.15.146:0/4041614495 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0af402ff60 0x7f0af4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:04.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:04.556+0000 7f0afdffb700 1 -- 172.21.15.146:0/4041614495 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0b040410f0 con 0x7f0b080601c0 2024-01-21T20:01:04.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:04.557+0000 7f0b0c88b700 1 --2- 172.21.15.146:0/4041614495 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0af402ff60 0x7f0af4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:04.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:04.557+0000 7f0b0c88b700 1 --2- 172.21.15.146:0/4041614495 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0af402ff60 0x7f0af4032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f0af80096a0 tx=0x7f0af8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:04.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:04.561+0000 7f0afdffb700 1 -- 172.21.15.146:0/4041614495 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0b040126f0 con 0x7f0b080601c0 2024-01-21T20:01:04.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:04.764+0000 7f0b0f2f0700 1 -- 172.21.15.146:0/4041614495 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0b080fbf50 con 0x7f0b080601c0 2024-01-21T20:01:04.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:04.765+0000 7f0afdffb700 1 -- 172.21.15.146:0/4041614495 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0b04012330 con 0x7f0b080601c0 2024-01-21T20:01:04.756 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:01:04.757 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:01:04.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:04.766+0000 7f0b0f2f0700 1 -- 172.21.15.146:0/4041614495 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0af402ff60 msgr2=0x7f0af4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:04.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:04.766+0000 7f0b0f2f0700 1 --2- 172.21.15.146:0/4041614495 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0af402ff60 0x7f0af4032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f0af80096a0 tx=0x7f0af8006b40).stop 2024-01-21T20:01:04.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:04.766+0000 7f0b0f2f0700 1 -- 172.21.15.146:0/4041614495 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0b080601c0 msgr2=0x7f0b080f6c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:04.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:04.766+0000 7f0b0f2f0700 1 --2- 172.21.15.146:0/4041614495 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0b080601c0 0x7f0b080f6c30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f0b04013510 tx=0x7f0b0400b040).stop 2024-01-21T20:01:04.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:04.766+0000 7f0b0f2f0700 1 -- 172.21.15.146:0/4041614495 shutdown_connections 2024-01-21T20:01:04.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:04.766+0000 7f0b0f2f0700 1 --2- 172.21.15.146:0/4041614495 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0af402ff60 0x7f0af4032420 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:04.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:04.766+0000 7f0b0f2f0700 1 --2- 172.21.15.146:0/4041614495 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0b080601c0 0x7f0b080f6c30 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:04.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:04.766+0000 7f0b0f2f0700 1 -- 172.21.15.146:0/4041614495 >> 172.21.15.146:0/4041614495 conn(0x7f0b080f0b70 msgr2=0x7f0b080f1690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:04.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:04.766+0000 7f0b0f2f0700 1 -- 172.21.15.146:0/4041614495 shutdown_connections 2024-01-21T20:01:04.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:04.766+0000 7f0b0f2f0700 1 -- 172.21.15.146:0/4041614495 wait complete. 2024-01-21T20:01:04.759 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:01:04.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:04 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:01:03.142037+0000 mgr.smithi053.fgsmol (mgr.14164) 58 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:05.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:05 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:01:04.765877+0000 mon.smithi053 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.146:0/4041614495' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:01:06.182 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:01:06.182 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:01:06.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:06 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:01:05.142457+0000 mgr.smithi053.fgsmol (mgr.14164) 59 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:07.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:07.416+0000 7f3c5f59e700 1 -- 172.21.15.146:0/1801553048 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3c600f62a0 msgr2=0x7f3c600f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:07.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:07.416+0000 7f3c5f59e700 1 --2- 172.21.15.146:0/1801553048 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3c600f62a0 0x7f3c600f66c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f3c50009fb0 tx=0x7f3c50009b30).stop 2024-01-21T20:01:07.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:07.416+0000 7f3c5f59e700 1 -- 172.21.15.146:0/1801553048 shutdown_connections 2024-01-21T20:01:07.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:07.416+0000 7f3c5f59e700 1 --2- 172.21.15.146:0/1801553048 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3c600f62a0 0x7f3c600f66c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:07.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:07.416+0000 7f3c5f59e700 1 -- 172.21.15.146:0/1801553048 >> 172.21.15.146:0/1801553048 conn(0x7f3c600f1aa0 msgr2=0x7f3c600f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:07.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:07.416+0000 7f3c5f59e700 1 -- 172.21.15.146:0/1801553048 shutdown_connections 2024-01-21T20:01:07.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:07.416+0000 7f3c5f59e700 1 -- 172.21.15.146:0/1801553048 wait complete. 2024-01-21T20:01:07.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:07.417+0000 7f3c5f59e700 1 Processor -- start 2024-01-21T20:01:07.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:07.417+0000 7f3c5f59e700 1 -- start start 2024-01-21T20:01:07.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:07.417+0000 7f3c5f59e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3c600f62a0 0x7f3c601064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:07.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:07.417+0000 7f3c5f59e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c601069a0 con 0x7f3c600f62a0 2024-01-21T20:01:07.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:07.417+0000 7f3c5e59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3c600f62a0 0x7f3c601064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:07.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:07.417+0000 7f3c5e59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3c600f62a0 0x7f3c601064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:54194/0 (socket says 172.21.15.146:54194) 2024-01-21T20:01:07.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:07.417+0000 7f3c5e59c700 1 -- 172.21.15.146:0/1614590369 learned_addr learned my addr 172.21.15.146:0/1614590369 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:01:07.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:07.418+0000 7f3c5e59c700 1 -- 172.21.15.146:0/1614590369 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c50005040 con 0x7f3c600f62a0 2024-01-21T20:01:07.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:07.418+0000 7f3c5e59c700 1 --2- 172.21.15.146:0/1614590369 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3c600f62a0 0x7f3c601064b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f3c50013510 tx=0x7f3c5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:07.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:07.418+0000 7f3c4effd700 1 -- 172.21.15.146:0/1614590369 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c50009320 con 0x7f3c600f62a0 2024-01-21T20:01:07.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:07.418+0000 7f3c5f59e700 1 -- 172.21.15.146:0/1614590369 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c60106ba0 con 0x7f3c600f62a0 2024-01-21T20:01:07.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:07.418+0000 7f3c4effd700 1 -- 172.21.15.146:0/1614590369 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3c50013f00 con 0x7f3c600f62a0 2024-01-21T20:01:07.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:07.418+0000 7f3c4effd700 1 -- 172.21.15.146:0/1614590369 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c50014dc0 con 0x7f3c600f62a0 2024-01-21T20:01:07.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:07.418+0000 7f3c5f59e700 1 -- 172.21.15.146:0/1614590369 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c60107030 con 0x7f3c600f62a0 2024-01-21T20:01:07.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:07.419+0000 7f3c4effd700 1 -- 172.21.15.146:0/1614590369 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3c500095f0 con 0x7f3c600f62a0 2024-01-21T20:01:07.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:07.419+0000 7f3c5f59e700 1 -- 172.21.15.146:0/1614590369 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c60045ad0 con 0x7f3c600f62a0 2024-01-21T20:01:07.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:07.419+0000 7f3c4effd700 1 --2- 172.21.15.146:0/1614590369 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f3c4802ff60 0x7f3c48032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:07.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:07.419+0000 7f3c4effd700 1 -- 172.21.15.146:0/1614590369 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3c50041000 con 0x7f3c600f62a0 2024-01-21T20:01:07.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:07.419+0000 7f3c5dd9b700 1 --2- 172.21.15.146:0/1614590369 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f3c4802ff60 0x7f3c48032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:07.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:07.419+0000 7f3c5dd9b700 1 --2- 172.21.15.146:0/1614590369 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f3c4802ff60 0x7f3c48032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f3c540096a0 tx=0x7f3c54006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:07.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:07.424+0000 7f3c4effd700 1 -- 172.21.15.146:0/1614590369 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3c50014420 con 0x7f3c600f62a0 2024-01-21T20:01:07.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:07.630+0000 7f3c5f59e700 1 -- 172.21.15.146:0/1614590369 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3c60107ee0 con 0x7f3c600f62a0 2024-01-21T20:01:07.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:07.630+0000 7f3c4effd700 1 -- 172.21.15.146:0/1614590369 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3c50010230 con 0x7f3c600f62a0 2024-01-21T20:01:07.622 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:01:07.622 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:01:07.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:07.632+0000 7f3c5f59e700 1 -- 172.21.15.146:0/1614590369 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f3c4802ff60 msgr2=0x7f3c48032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:07.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:07.632+0000 7f3c5f59e700 1 --2- 172.21.15.146:0/1614590369 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f3c4802ff60 0x7f3c48032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f3c540096a0 tx=0x7f3c54006b40).stop 2024-01-21T20:01:07.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:07.632+0000 7f3c5f59e700 1 -- 172.21.15.146:0/1614590369 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3c600f62a0 msgr2=0x7f3c601064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:07.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:07.632+0000 7f3c5f59e700 1 --2- 172.21.15.146:0/1614590369 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3c600f62a0 0x7f3c601064b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f3c50013510 tx=0x7f3c5000b040).stop 2024-01-21T20:01:07.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:07.632+0000 7f3c5f59e700 1 -- 172.21.15.146:0/1614590369 shutdown_connections 2024-01-21T20:01:07.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:07.632+0000 7f3c5f59e700 1 --2- 172.21.15.146:0/1614590369 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f3c4802ff60 0x7f3c48032420 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:07.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:07.632+0000 7f3c5f59e700 1 --2- 172.21.15.146:0/1614590369 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3c600f62a0 0x7f3c601064b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:07.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:07.632+0000 7f3c5f59e700 1 -- 172.21.15.146:0/1614590369 >> 172.21.15.146:0/1614590369 conn(0x7f3c600f1aa0 msgr2=0x7f3c600f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:07.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:07.632+0000 7f3c5f59e700 1 -- 172.21.15.146:0/1614590369 shutdown_connections 2024-01-21T20:01:07.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:07.632+0000 7f3c5f59e700 1 -- 172.21.15.146:0/1614590369 wait complete. 2024-01-21T20:01:07.624 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:01:08.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:08 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:01:07.142865+0000 mgr.smithi053.fgsmol (mgr.14164) 60 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:08.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:08 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:01:07.631762+0000 mon.smithi053 (mon.0) 150 : audit [DBG] from='client.? 172.21.15.146:0/1614590369' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:01:09.020 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:01:09.020 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:01:09.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:09 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:01:09.263919+0000 mon.smithi053 (mon.0) 151 : audit [INF] from='mgr.14164 172.21.15.53:0/2331671680' entity='mgr.smithi053.fgsmol' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fgsmol/mirror_snapshot_schedule"}]: dispatch 2024-01-21T20:01:09.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:09 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:01:09.272823+0000 mon.smithi053 (mon.0) 152 : audit [INF] from='mgr.14164 172.21.15.53:0/2331671680' entity='mgr.smithi053.fgsmol' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fgsmol/trash_purge_schedule"}]: dispatch 2024-01-21T20:01:10.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:10.235+0000 7f0000a3f700 1 -- 172.21.15.146:0/4105259250 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efffc0f59d0 msgr2=0x7efffc0f5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:10.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:10.235+0000 7f0000a3f700 1 --2- 172.21.15.146:0/4105259250 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efffc0f59d0 0x7efffc0f5df0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7effec009fb0 tx=0x7effec009b30).stop 2024-01-21T20:01:10.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:10.235+0000 7f0000a3f700 1 -- 172.21.15.146:0/4105259250 shutdown_connections 2024-01-21T20:01:10.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:10.235+0000 7f0000a3f700 1 --2- 172.21.15.146:0/4105259250 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efffc0f59d0 0x7efffc0f5df0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:10.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:10.235+0000 7f0000a3f700 1 -- 172.21.15.146:0/4105259250 >> 172.21.15.146:0/4105259250 conn(0x7efffc0f02e0 msgr2=0x7efffc0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:10.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:10.235+0000 7f0000a3f700 1 -- 172.21.15.146:0/4105259250 shutdown_connections 2024-01-21T20:01:10.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:10.235+0000 7f0000a3f700 1 -- 172.21.15.146:0/4105259250 wait complete. 2024-01-21T20:01:10.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:10.235+0000 7f0000a3f700 1 Processor -- start 2024-01-21T20:01:10.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:10.235+0000 7f0000a3f700 1 -- start start 2024-01-21T20:01:10.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:10.235+0000 7f0000a3f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efffc0f59d0 0x7efffc1093e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:10.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:10.235+0000 7f0000a3f700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efffc0f5390 con 0x7efffc0f59d0 2024-01-21T20:01:10.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:10.236+0000 7efffa59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efffc0f59d0 0x7efffc1093e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:10.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:10.236+0000 7efffa59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efffc0f59d0 0x7efffc1093e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:54224/0 (socket says 172.21.15.146:54224) 2024-01-21T20:01:10.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:10.236+0000 7efffa59c700 1 -- 172.21.15.146:0/2722943067 learned_addr learned my addr 172.21.15.146:0/2722943067 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:01:10.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:10.236+0000 7efffa59c700 1 -- 172.21.15.146:0/2722943067 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effec005040 con 0x7efffc0f59d0 2024-01-21T20:01:10.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:10.236+0000 7efffa59c700 1 --2- 172.21.15.146:0/2722943067 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efffc0f59d0 0x7efffc1093e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7effec014000 tx=0x7effec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:10.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:10.236+0000 7effeaffd700 1 -- 172.21.15.146:0/2722943067 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effec0095f0 con 0x7efffc0f59d0 2024-01-21T20:01:10.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:10.236+0000 7f0000a3f700 1 -- 172.21.15.146:0/2722943067 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efffc0f5590 con 0x7efffc0f59d0 2024-01-21T20:01:10.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:10.236+0000 7effeaffd700 1 -- 172.21.15.146:0/2722943067 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7effec007e70 con 0x7efffc0f59d0 2024-01-21T20:01:10.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:10.236+0000 7effeaffd700 1 -- 172.21.15.146:0/2722943067 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effec014ea0 con 0x7efffc0f59d0 2024-01-21T20:01:10.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:10.236+0000 7f0000a3f700 1 -- 172.21.15.146:0/2722943067 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efffc109a90 con 0x7efffc0f59d0 2024-01-21T20:01:10.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:10.237+0000 7effeaffd700 1 -- 172.21.15.146:0/2722943067 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7effec025070 con 0x7efffc0f59d0 2024-01-21T20:01:10.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:10.237+0000 7f0000a3f700 1 -- 172.21.15.146:0/2722943067 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efffc059100 con 0x7efffc0f59d0 2024-01-21T20:01:10.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:10.237+0000 7effeaffd700 1 --2- 172.21.15.146:0/2722943067 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7effe402ff60 0x7effe4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:10.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:10.237+0000 7effeaffd700 1 -- 172.21.15.146:0/2722943067 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7effec0409b0 con 0x7efffc0f59d0 2024-01-21T20:01:10.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:10.238+0000 7efff9d9b700 1 --2- 172.21.15.146:0/2722943067 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7effe402ff60 0x7effe4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:10.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:10.238+0000 7efff9d9b700 1 --2- 172.21.15.146:0/2722943067 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7effe402ff60 0x7effe4032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7efff00096a0 tx=0x7efff0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:10.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:10.242+0000 7effeaffd700 1 -- 172.21.15.146:0/2722943067 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7effec026350 con 0x7efffc0f59d0 2024-01-21T20:01:10.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:10.449+0000 7f0000a3f700 1 -- 172.21.15.146:0/2722943067 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efffc0fae50 con 0x7efffc0f59d0 2024-01-21T20:01:10.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:10.449+0000 7effeaffd700 1 -- 172.21.15.146:0/2722943067 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7effec025300 con 0x7efffc0f59d0 2024-01-21T20:01:10.441 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:01:10.441 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:01:10.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:10.451+0000 7f0000a3f700 1 -- 172.21.15.146:0/2722943067 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7effe402ff60 msgr2=0x7effe4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:10.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:10.451+0000 7f0000a3f700 1 --2- 172.21.15.146:0/2722943067 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7effe402ff60 0x7effe4032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7efff00096a0 tx=0x7efff0006b40).stop 2024-01-21T20:01:10.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:10.451+0000 7f0000a3f700 1 -- 172.21.15.146:0/2722943067 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efffc0f59d0 msgr2=0x7efffc1093e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:10.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:10.451+0000 7f0000a3f700 1 --2- 172.21.15.146:0/2722943067 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efffc0f59d0 0x7efffc1093e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7effec014000 tx=0x7effec00b040).stop 2024-01-21T20:01:10.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:10.451+0000 7f0000a3f700 1 -- 172.21.15.146:0/2722943067 shutdown_connections 2024-01-21T20:01:10.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:10.451+0000 7f0000a3f700 1 --2- 172.21.15.146:0/2722943067 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7effe402ff60 0x7effe4032420 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:10.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:10.451+0000 7f0000a3f700 1 --2- 172.21.15.146:0/2722943067 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efffc0f59d0 0x7efffc1093e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:10.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:10.451+0000 7f0000a3f700 1 -- 172.21.15.146:0/2722943067 >> 172.21.15.146:0/2722943067 conn(0x7efffc0f02e0 msgr2=0x7efffc0f0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:10.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:10.451+0000 7f0000a3f700 1 -- 172.21.15.146:0/2722943067 shutdown_connections 2024-01-21T20:01:10.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:10.451+0000 7f0000a3f700 1 -- 172.21.15.146:0/2722943067 wait complete. 2024-01-21T20:01:10.444 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:01:10.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:10 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:01:09.143196+0000 mgr.smithi053.fgsmol (mgr.14164) 61 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:10.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:10 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:01:10.450675+0000 mon.smithi053 (mon.0) 153 : audit [DBG] from='client.? 172.21.15.146:0/2722943067' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:01:11.867 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:01:11.868 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:01:12.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:12 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:01:11.143486+0000 mgr.smithi053.fgsmol (mgr.14164) 62 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:13.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:13.070+0000 7f3e9ccee700 1 -- 172.21.15.146:0/2098504143 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e980f5410 msgr2=0x7f3e980f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:13.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:13.070+0000 7f3e9ccee700 1 --2- 172.21.15.146:0/2098504143 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e980f5410 0x7f3e980f5830 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f3e8c009fb0 tx=0x7f3e8c009b30).stop 2024-01-21T20:01:13.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:13.070+0000 7f3e9ccee700 1 -- 172.21.15.146:0/2098504143 shutdown_connections 2024-01-21T20:01:13.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:13.070+0000 7f3e9ccee700 1 --2- 172.21.15.146:0/2098504143 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e980f5410 0x7f3e980f5830 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:13.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:13.070+0000 7f3e9ccee700 1 -- 172.21.15.146:0/2098504143 >> 172.21.15.146:0/2098504143 conn(0x7f3e980f0b70 msgr2=0x7f3e980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:13.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:13.070+0000 7f3e9ccee700 1 -- 172.21.15.146:0/2098504143 shutdown_connections 2024-01-21T20:01:13.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:13.070+0000 7f3e9ccee700 1 -- 172.21.15.146:0/2098504143 wait complete. 2024-01-21T20:01:13.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:13.071+0000 7f3e9ccee700 1 Processor -- start 2024-01-21T20:01:13.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:13.071+0000 7f3e9ccee700 1 -- start start 2024-01-21T20:01:13.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:13.071+0000 7f3e9ccee700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e980f5410 0x7f3e9810b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:13.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:13.071+0000 7f3e9ccee700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e98107240 con 0x7f3e980f5410 2024-01-21T20:01:13.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:13.071+0000 7f3e9659c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e980f5410 0x7f3e9810b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:13.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:13.071+0000 7f3e9659c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e980f5410 0x7f3e9810b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:54250/0 (socket says 172.21.15.146:54250) 2024-01-21T20:01:13.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:13.071+0000 7f3e9659c700 1 -- 172.21.15.146:0/1214391593 learned_addr learned my addr 172.21.15.146:0/1214391593 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:01:13.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:13.071+0000 7f3e9659c700 1 -- 172.21.15.146:0/1214391593 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e8c005040 con 0x7f3e980f5410 2024-01-21T20:01:13.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:13.072+0000 7f3e9659c700 1 --2- 172.21.15.146:0/1214391593 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e980f5410 0x7f3e9810b850 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f3e8c013510 tx=0x7f3e8c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:13.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:13.072+0000 7f3e86ffd700 1 -- 172.21.15.146:0/1214391593 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e8c009320 con 0x7f3e980f5410 2024-01-21T20:01:13.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:13.072+0000 7f3e9ccee700 1 -- 172.21.15.146:0/1214391593 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e98060220 con 0x7f3e980f5410 2024-01-21T20:01:13.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:13.072+0000 7f3e86ffd700 1 -- 172.21.15.146:0/1214391593 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3e8c013f00 con 0x7f3e980f5410 2024-01-21T20:01:13.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:13.072+0000 7f3e86ffd700 1 -- 172.21.15.146:0/1214391593 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e8c014d90 con 0x7f3e980f5410 2024-01-21T20:01:13.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:13.072+0000 7f3e9ccee700 1 -- 172.21.15.146:0/1214391593 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e9810bd40 con 0x7f3e980f5410 2024-01-21T20:01:13.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:13.073+0000 7f3e9ccee700 1 -- 172.21.15.146:0/1214391593 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e98045ad0 con 0x7f3e980f5410 2024-01-21T20:01:13.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:13.073+0000 7f3e86ffd700 1 -- 172.21.15.146:0/1214391593 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3e8c0095f0 con 0x7f3e980f5410 2024-01-21T20:01:13.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:13.073+0000 7f3e86ffd700 1 --2- 172.21.15.146:0/1214391593 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f3e8002ff60 0x7f3e80032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:13.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:13.073+0000 7f3e86ffd700 1 -- 172.21.15.146:0/1214391593 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3e8c040fc0 con 0x7f3e980f5410 2024-01-21T20:01:13.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:13.073+0000 7f3e95d9b700 1 --2- 172.21.15.146:0/1214391593 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f3e8002ff60 0x7f3e80032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:13.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:13.074+0000 7f3e95d9b700 1 --2- 172.21.15.146:0/1214391593 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f3e8002ff60 0x7f3e80032420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f3e880096a0 tx=0x7f3e88006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:13.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:13.078+0000 7f3e86ffd700 1 -- 172.21.15.146:0/1214391593 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3e8c014420 con 0x7f3e980f5410 2024-01-21T20:01:13.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:13.285+0000 7f3e9ccee700 1 -- 172.21.15.146:0/1214391593 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3e980fae60 con 0x7f3e980f5410 2024-01-21T20:01:13.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:13.285+0000 7f3e86ffd700 1 -- 172.21.15.146:0/1214391593 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3e8c010230 con 0x7f3e980f5410 2024-01-21T20:01:13.277 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:01:13.277 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:01:13.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:13.287+0000 7f3e9ccee700 1 -- 172.21.15.146:0/1214391593 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f3e8002ff60 msgr2=0x7f3e80032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:13.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:13.287+0000 7f3e9ccee700 1 --2- 172.21.15.146:0/1214391593 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f3e8002ff60 0x7f3e80032420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f3e880096a0 tx=0x7f3e88006b40).stop 2024-01-21T20:01:13.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:13.287+0000 7f3e9ccee700 1 -- 172.21.15.146:0/1214391593 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e980f5410 msgr2=0x7f3e9810b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:13.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:13.287+0000 7f3e9ccee700 1 --2- 172.21.15.146:0/1214391593 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e980f5410 0x7f3e9810b850 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f3e8c013510 tx=0x7f3e8c00b040).stop 2024-01-21T20:01:13.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:13.287+0000 7f3e9ccee700 1 -- 172.21.15.146:0/1214391593 shutdown_connections 2024-01-21T20:01:13.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:13.287+0000 7f3e9ccee700 1 --2- 172.21.15.146:0/1214391593 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f3e8002ff60 0x7f3e80032420 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:13.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:13.287+0000 7f3e9ccee700 1 --2- 172.21.15.146:0/1214391593 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e980f5410 0x7f3e9810b850 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:13.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:13.287+0000 7f3e9ccee700 1 -- 172.21.15.146:0/1214391593 >> 172.21.15.146:0/1214391593 conn(0x7f3e980f0b70 msgr2=0x7f3e980f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:13.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:13.287+0000 7f3e9ccee700 1 -- 172.21.15.146:0/1214391593 shutdown_connections 2024-01-21T20:01:13.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:13.287+0000 7f3e9ccee700 1 -- 172.21.15.146:0/1214391593 wait complete. 2024-01-21T20:01:13.279 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:01:13.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:13 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:01:13.286697+0000 mon.smithi053 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.146:0/1214391593' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:01:14.697 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:01:14.698 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:01:14.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:14 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:01:13.143835+0000 mgr.smithi053.fgsmol (mgr.14164) 63 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:15.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:15.945+0000 7fe999dc4700 1 -- 172.21.15.146:0/3648576493 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9940f6510 msgr2=0x7fe9940f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:15.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:15.945+0000 7fe999dc4700 1 --2- 172.21.15.146:0/3648576493 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9940f6510 0x7fe9940f6930 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fe97c009fb0 tx=0x7fe97c009b30).stop 2024-01-21T20:01:15.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:15.945+0000 7fe999dc4700 1 -- 172.21.15.146:0/3648576493 shutdown_connections 2024-01-21T20:01:15.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:15.945+0000 7fe999dc4700 1 --2- 172.21.15.146:0/3648576493 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9940f6510 0x7fe9940f6930 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:15.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:15.945+0000 7fe999dc4700 1 -- 172.21.15.146:0/3648576493 >> 172.21.15.146:0/3648576493 conn(0x7fe9940f1c70 msgr2=0x7fe9940f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:15.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:15.945+0000 7fe999dc4700 1 -- 172.21.15.146:0/3648576493 shutdown_connections 2024-01-21T20:01:15.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:15.945+0000 7fe999dc4700 1 -- 172.21.15.146:0/3648576493 wait complete. 2024-01-21T20:01:15.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:15.946+0000 7fe999dc4700 1 Processor -- start 2024-01-21T20:01:15.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:15.946+0000 7fe999dc4700 1 -- start start 2024-01-21T20:01:15.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:15.946+0000 7fe999dc4700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9940f6510 0x7fe994106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:15.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:15.946+0000 7fe999dc4700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe994106c20 con 0x7fe9940f6510 2024-01-21T20:01:15.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:15.946+0000 7fe9937fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9940f6510 0x7fe994106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:15.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:15.946+0000 7fe9937fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9940f6510 0x7fe994106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:54276/0 (socket says 172.21.15.146:54276) 2024-01-21T20:01:15.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:15.946+0000 7fe9937fe700 1 -- 172.21.15.146:0/1077278345 learned_addr learned my addr 172.21.15.146:0/1077278345 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:01:15.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:15.947+0000 7fe9937fe700 1 -- 172.21.15.146:0/1077278345 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe97c005040 con 0x7fe9940f6510 2024-01-21T20:01:15.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:15.947+0000 7fe9937fe700 1 --2- 172.21.15.146:0/1077278345 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9940f6510 0x7fe994106730 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fe97c013510 tx=0x7fe97c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:15.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:15.947+0000 7fe98bfff700 1 -- 172.21.15.146:0/1077278345 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe97c009320 con 0x7fe9940f6510 2024-01-21T20:01:15.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:15.947+0000 7fe999dc4700 1 -- 172.21.15.146:0/1077278345 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe994106e20 con 0x7fe9940f6510 2024-01-21T20:01:15.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:15.947+0000 7fe98bfff700 1 -- 172.21.15.146:0/1077278345 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe97c013f00 con 0x7fe9940f6510 2024-01-21T20:01:15.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:15.947+0000 7fe98bfff700 1 -- 172.21.15.146:0/1077278345 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe97c014d90 con 0x7fe9940f6510 2024-01-21T20:01:15.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:15.947+0000 7fe999dc4700 1 -- 172.21.15.146:0/1077278345 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9941072b0 con 0x7fe9940f6510 2024-01-21T20:01:15.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:15.948+0000 7fe999dc4700 1 -- 172.21.15.146:0/1077278345 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe994045ad0 con 0x7fe9940f6510 2024-01-21T20:01:15.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:15.948+0000 7fe98bfff700 1 -- 172.21.15.146:0/1077278345 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe97c0095f0 con 0x7fe9940f6510 2024-01-21T20:01:15.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:15.948+0000 7fe98bfff700 1 --2- 172.21.15.146:0/1077278345 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fe98002ff60 0x7fe980032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:15.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:15.948+0000 7fe98bfff700 1 -- 172.21.15.146:0/1077278345 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe97c041030 con 0x7fe9940f6510 2024-01-21T20:01:15.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:15.948+0000 7fe992ffd700 1 --2- 172.21.15.146:0/1077278345 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fe98002ff60 0x7fe980032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:15.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:15.949+0000 7fe992ffd700 1 --2- 172.21.15.146:0/1077278345 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fe98002ff60 0x7fe980032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fe9840096a0 tx=0x7fe984006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:15.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:15.953+0000 7fe98bfff700 1 -- 172.21.15.146:0/1077278345 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe97c014420 con 0x7fe9940f6510 2024-01-21T20:01:16.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:16.158+0000 7fe999dc4700 1 -- 172.21.15.146:0/1077278345 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe9940fbf60 con 0x7fe9940f6510 2024-01-21T20:01:16.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:16.159+0000 7fe98bfff700 1 -- 172.21.15.146:0/1077278345 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe97c010230 con 0x7fe9940f6510 2024-01-21T20:01:16.150 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:01:16.150 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:01:16.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:16.160+0000 7fe999dc4700 1 -- 172.21.15.146:0/1077278345 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fe98002ff60 msgr2=0x7fe980032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:16.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:16.160+0000 7fe999dc4700 1 --2- 172.21.15.146:0/1077278345 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fe98002ff60 0x7fe980032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fe9840096a0 tx=0x7fe984006b40).stop 2024-01-21T20:01:16.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:16.160+0000 7fe999dc4700 1 -- 172.21.15.146:0/1077278345 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9940f6510 msgr2=0x7fe994106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:16.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:16.160+0000 7fe999dc4700 1 --2- 172.21.15.146:0/1077278345 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9940f6510 0x7fe994106730 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fe97c013510 tx=0x7fe97c00b040).stop 2024-01-21T20:01:16.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:16.160+0000 7fe999dc4700 1 -- 172.21.15.146:0/1077278345 shutdown_connections 2024-01-21T20:01:16.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:16.160+0000 7fe999dc4700 1 --2- 172.21.15.146:0/1077278345 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fe98002ff60 0x7fe980032420 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:16.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:16.160+0000 7fe999dc4700 1 --2- 172.21.15.146:0/1077278345 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9940f6510 0x7fe994106730 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:16.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:16.160+0000 7fe999dc4700 1 -- 172.21.15.146:0/1077278345 >> 172.21.15.146:0/1077278345 conn(0x7fe9940f1c70 msgr2=0x7fe9940f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:16.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:16.160+0000 7fe999dc4700 1 -- 172.21.15.146:0/1077278345 shutdown_connections 2024-01-21T20:01:16.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:16.160+0000 7fe999dc4700 1 -- 172.21.15.146:0/1077278345 wait complete. 2024-01-21T20:01:16.153 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:01:16.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:01:15.144072+0000 mgr.smithi053.fgsmol (mgr.14164) 64 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:16.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:01:16.159789+0000 mon.smithi053 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.146:0/1077278345' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:01:17.559 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:01:17.560 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:01:18.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:18.759+0000 7f71524e9700 1 -- 172.21.15.146:0/1463141988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f714c0f51a0 msgr2=0x7f714c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:18.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:18.759+0000 7f71524e9700 1 --2- 172.21.15.146:0/1463141988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f714c0f51a0 0x7f714c0f55c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f7148009fb0 tx=0x7f7148009b30).stop 2024-01-21T20:01:18.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:18.759+0000 7f71524e9700 1 -- 172.21.15.146:0/1463141988 shutdown_connections 2024-01-21T20:01:18.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:18.759+0000 7f71524e9700 1 --2- 172.21.15.146:0/1463141988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f714c0f51a0 0x7f714c0f55c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:18.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:18.759+0000 7f71524e9700 1 -- 172.21.15.146:0/1463141988 >> 172.21.15.146:0/1463141988 conn(0x7f714c0f09a0 msgr2=0x7f714c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:18.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:18.759+0000 7f71524e9700 1 -- 172.21.15.146:0/1463141988 shutdown_connections 2024-01-21T20:01:18.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:18.759+0000 7f71524e9700 1 -- 172.21.15.146:0/1463141988 wait complete. 2024-01-21T20:01:18.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:18.759+0000 7f71524e9700 1 Processor -- start 2024-01-21T20:01:18.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:18.759+0000 7f71524e9700 1 -- start start 2024-01-21T20:01:18.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:18.760+0000 7f71524e9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f714c0f51a0 0x7f714c1091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:18.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:18.760+0000 7f71524e9700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714c060240 con 0x7f714c0f51a0 2024-01-21T20:01:18.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:18.760+0000 7f71514e7700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f714c0f51a0 0x7f714c1091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:18.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:18.760+0000 7f71514e7700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f714c0f51a0 0x7f714c1091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:54744/0 (socket says 172.21.15.146:54744) 2024-01-21T20:01:18.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:18.760+0000 7f71514e7700 1 -- 172.21.15.146:0/4214512180 learned_addr learned my addr 172.21.15.146:0/4214512180 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:01:18.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:18.760+0000 7f71514e7700 1 -- 172.21.15.146:0/4214512180 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7148005040 con 0x7f714c0f51a0 2024-01-21T20:01:18.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:18.760+0000 7f71514e7700 1 --2- 172.21.15.146:0/4214512180 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f714c0f51a0 0x7f714c1091b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f7148013510 tx=0x7f714800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:18.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:18.760+0000 7f7141ffb700 1 -- 172.21.15.146:0/4214512180 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7148009320 con 0x7f714c0f51a0 2024-01-21T20:01:18.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:18.760+0000 7f71524e9700 1 -- 172.21.15.146:0/4214512180 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f714c060440 con 0x7f714c0f51a0 2024-01-21T20:01:18.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:18.760+0000 7f7141ffb700 1 -- 172.21.15.146:0/4214512180 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7148013f00 con 0x7f714c0f51a0 2024-01-21T20:01:18.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:18.760+0000 7f7141ffb700 1 -- 172.21.15.146:0/4214512180 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7148014dc0 con 0x7f714c0f51a0 2024-01-21T20:01:18.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:18.761+0000 7f71524e9700 1 -- 172.21.15.146:0/4214512180 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f714c109860 con 0x7f714c0f51a0 2024-01-21T20:01:18.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:18.762+0000 7f7141ffb700 1 -- 172.21.15.146:0/4214512180 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7148014420 con 0x7f714c0f51a0 2024-01-21T20:01:18.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:18.762+0000 7f71524e9700 1 -- 172.21.15.146:0/4214512180 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f714c045ad0 con 0x7f714c0f51a0 2024-01-21T20:01:18.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:18.762+0000 7f7141ffb700 1 --2- 172.21.15.146:0/4214512180 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f713802ff60 0x7f7138032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:18.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:18.762+0000 7f7141ffb700 1 -- 172.21.15.146:0/4214512180 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7148041020 con 0x7f714c0f51a0 2024-01-21T20:01:18.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:18.762+0000 7f7150ce6700 1 --2- 172.21.15.146:0/4214512180 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f713802ff60 0x7f7138032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:18.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:18.762+0000 7f7150ce6700 1 --2- 172.21.15.146:0/4214512180 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f713802ff60 0x7f7138032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f713c0096a0 tx=0x7f713c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:18.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:18.767+0000 7f7141ffb700 1 -- 172.21.15.146:0/4214512180 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f71480095f0 con 0x7f714c0f51a0 2024-01-21T20:01:18.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:18 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:01:17.144376+0000 mgr.smithi053.fgsmol (mgr.14164) 65 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:18.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:18.974+0000 7f71524e9700 1 -- 172.21.15.146:0/4214512180 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f714c10a080 con 0x7f714c0f51a0 2024-01-21T20:01:18.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:18.975+0000 7f7141ffb700 1 -- 172.21.15.146:0/4214512180 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f714801cb10 con 0x7f714c0f51a0 2024-01-21T20:01:18.967 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:01:18.967 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:01:18.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:18.977+0000 7f71524e9700 1 -- 172.21.15.146:0/4214512180 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f713802ff60 msgr2=0x7f7138032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:18.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:18.977+0000 7f71524e9700 1 --2- 172.21.15.146:0/4214512180 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f713802ff60 0x7f7138032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f713c0096a0 tx=0x7f713c006b40).stop 2024-01-21T20:01:18.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:18.977+0000 7f71524e9700 1 -- 172.21.15.146:0/4214512180 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f714c0f51a0 msgr2=0x7f714c1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:18.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:18.977+0000 7f71524e9700 1 --2- 172.21.15.146:0/4214512180 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f714c0f51a0 0x7f714c1091b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f7148013510 tx=0x7f714800b040).stop 2024-01-21T20:01:18.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:18.977+0000 7f71524e9700 1 -- 172.21.15.146:0/4214512180 shutdown_connections 2024-01-21T20:01:18.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:18.977+0000 7f71524e9700 1 --2- 172.21.15.146:0/4214512180 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f713802ff60 0x7f7138032420 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:18.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:18.977+0000 7f71524e9700 1 --2- 172.21.15.146:0/4214512180 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f714c0f51a0 0x7f714c1091b0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:18.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:18.977+0000 7f71524e9700 1 -- 172.21.15.146:0/4214512180 >> 172.21.15.146:0/4214512180 conn(0x7f714c0f09a0 msgr2=0x7f714c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:18.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:18.977+0000 7f71524e9700 1 -- 172.21.15.146:0/4214512180 shutdown_connections 2024-01-21T20:01:18.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:18.977+0000 7f71524e9700 1 -- 172.21.15.146:0/4214512180 wait complete. 2024-01-21T20:01:18.970 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:01:19.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:19 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:01:18.976538+0000 mon.smithi053 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.146:0/4214512180' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:01:20.398 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:01:20.398 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:01:20.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:20 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:01:19.144634+0000 mgr.smithi053.fgsmol (mgr.14164) 66 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:21.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:21.622+0000 7f0d60f47700 1 -- 172.21.15.146:0/3968963416 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d5c0f47c0 msgr2=0x7f0d5c0f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:21.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:21.622+0000 7f0d60f47700 1 --2- 172.21.15.146:0/3968963416 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d5c0f47c0 0x7f0d5c0f4be0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f0d44009fb0 tx=0x7f0d44009b30).stop 2024-01-21T20:01:21.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:21.622+0000 7f0d60f47700 1 -- 172.21.15.146:0/3968963416 shutdown_connections 2024-01-21T20:01:21.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:21.622+0000 7f0d60f47700 1 --2- 172.21.15.146:0/3968963416 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d5c0f47c0 0x7f0d5c0f4be0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:21.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:21.622+0000 7f0d60f47700 1 -- 172.21.15.146:0/3968963416 >> 172.21.15.146:0/3968963416 conn(0x7f0d5c0f02e0 msgr2=0x7f0d5c0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:21.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:21.622+0000 7f0d60f47700 1 -- 172.21.15.146:0/3968963416 shutdown_connections 2024-01-21T20:01:21.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:21.622+0000 7f0d60f47700 1 -- 172.21.15.146:0/3968963416 wait complete. 2024-01-21T20:01:21.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:21.622+0000 7f0d60f47700 1 Processor -- start 2024-01-21T20:01:21.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:21.622+0000 7f0d60f47700 1 -- start start 2024-01-21T20:01:21.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:21.622+0000 7f0d60f47700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d5c0f47c0 0x7f0d5c106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:21.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:21.622+0000 7f0d60f47700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5c106bf0 con 0x7f0d5c0f47c0 2024-01-21T20:01:21.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:21.623+0000 7f0d5a59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d5c0f47c0 0x7f0d5c106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:21.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:21.623+0000 7f0d5a59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d5c0f47c0 0x7f0d5c106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:54772/0 (socket says 172.21.15.146:54772) 2024-01-21T20:01:21.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:21.623+0000 7f0d5a59c700 1 -- 172.21.15.146:0/988194093 learned_addr learned my addr 172.21.15.146:0/988194093 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:01:21.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:21.623+0000 7f0d5a59c700 1 -- 172.21.15.146:0/988194093 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d44005040 con 0x7f0d5c0f47c0 2024-01-21T20:01:21.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:21.623+0000 7f0d5a59c700 1 --2- 172.21.15.146:0/988194093 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d5c0f47c0 0x7f0d5c106700 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f0d44013510 tx=0x7f0d4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:21.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:21.623+0000 7f0d52ffd700 1 -- 172.21.15.146:0/988194093 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d44009320 con 0x7f0d5c0f47c0 2024-01-21T20:01:21.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:21.623+0000 7f0d60f47700 1 -- 172.21.15.146:0/988194093 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d5c106df0 con 0x7f0d5c0f47c0 2024-01-21T20:01:21.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:21.623+0000 7f0d52ffd700 1 -- 172.21.15.146:0/988194093 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d44013f00 con 0x7f0d5c0f47c0 2024-01-21T20:01:21.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:21.623+0000 7f0d52ffd700 1 -- 172.21.15.146:0/988194093 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d44014d90 con 0x7f0d5c0f47c0 2024-01-21T20:01:21.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:21.623+0000 7f0d60f47700 1 -- 172.21.15.146:0/988194093 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d5c107280 con 0x7f0d5c0f47c0 2024-01-21T20:01:21.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:21.624+0000 7f0d60f47700 1 -- 172.21.15.146:0/988194093 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d5c045ad0 con 0x7f0d5c0f47c0 2024-01-21T20:01:21.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:21.624+0000 7f0d52ffd700 1 -- 172.21.15.146:0/988194093 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0d440095f0 con 0x7f0d5c0f47c0 2024-01-21T20:01:21.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:21.624+0000 7f0d52ffd700 1 --2- 172.21.15.146:0/988194093 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0d4802ff60 0x7f0d48032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:21.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:21.624+0000 7f0d52ffd700 1 -- 172.21.15.146:0/988194093 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0d44040fe0 con 0x7f0d5c0f47c0 2024-01-21T20:01:21.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:21.625+0000 7f0d59d9b700 1 --2- 172.21.15.146:0/988194093 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0d4802ff60 0x7f0d48032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:21.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:21.625+0000 7f0d59d9b700 1 --2- 172.21.15.146:0/988194093 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0d4802ff60 0x7f0d48032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f0d4c0096a0 tx=0x7f0d4c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:21.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:21.630+0000 7f0d52ffd700 1 -- 172.21.15.146:0/988194093 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0d44014420 con 0x7f0d5c0f47c0 2024-01-21T20:01:21.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:21.833+0000 7f0d60f47700 1 -- 172.21.15.146:0/988194093 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0d5c1080d0 con 0x7f0d5c0f47c0 2024-01-21T20:01:21.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:21.833+0000 7f0d52ffd700 1 -- 172.21.15.146:0/988194093 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0d44010230 con 0x7f0d5c0f47c0 2024-01-21T20:01:21.825 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:01:21.825 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:01:21.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:21.835+0000 7f0d60f47700 1 -- 172.21.15.146:0/988194093 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0d4802ff60 msgr2=0x7f0d48032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:21.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:21.835+0000 7f0d60f47700 1 --2- 172.21.15.146:0/988194093 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0d4802ff60 0x7f0d48032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f0d4c0096a0 tx=0x7f0d4c006b40).stop 2024-01-21T20:01:21.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:21.835+0000 7f0d60f47700 1 -- 172.21.15.146:0/988194093 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d5c0f47c0 msgr2=0x7f0d5c106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:21.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:21.835+0000 7f0d60f47700 1 --2- 172.21.15.146:0/988194093 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d5c0f47c0 0x7f0d5c106700 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f0d44013510 tx=0x7f0d4400b040).stop 2024-01-21T20:01:21.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:21.835+0000 7f0d60f47700 1 -- 172.21.15.146:0/988194093 shutdown_connections 2024-01-21T20:01:21.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:21.835+0000 7f0d60f47700 1 --2- 172.21.15.146:0/988194093 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0d4802ff60 0x7f0d48032420 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:21.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:21.835+0000 7f0d60f47700 1 --2- 172.21.15.146:0/988194093 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d5c0f47c0 0x7f0d5c106700 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:21.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:21.835+0000 7f0d60f47700 1 -- 172.21.15.146:0/988194093 >> 172.21.15.146:0/988194093 conn(0x7f0d5c0f02e0 msgr2=0x7f0d5c0f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:21.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:21.835+0000 7f0d60f47700 1 -- 172.21.15.146:0/988194093 shutdown_connections 2024-01-21T20:01:21.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:21.835+0000 7f0d60f47700 1 -- 172.21.15.146:0/988194093 wait complete. 2024-01-21T20:01:21.827 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:01:22.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:01:21.144955+0000 mgr.smithi053.fgsmol (mgr.14164) 67 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:22.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:01:21.834851+0000 mon.smithi053 (mon.0) 157 : audit [DBG] from='client.? 172.21.15.146:0/988194093' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:01:23.246 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:01:23.246 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:01:24.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:24.480+0000 7fbe1a031700 1 -- 172.21.15.146:0/4235778613 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe140f5410 msgr2=0x7fbe140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:24.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:24.480+0000 7fbe1a031700 1 --2- 172.21.15.146:0/4235778613 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe140f5410 0x7fbe140f5830 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fbe08009fb0 tx=0x7fbe08009b30).stop 2024-01-21T20:01:24.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:24.480+0000 7fbe1a031700 1 -- 172.21.15.146:0/4235778613 shutdown_connections 2024-01-21T20:01:24.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:24.480+0000 7fbe1a031700 1 --2- 172.21.15.146:0/4235778613 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe140f5410 0x7fbe140f5830 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:24.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:24.480+0000 7fbe1a031700 1 -- 172.21.15.146:0/4235778613 >> 172.21.15.146:0/4235778613 conn(0x7fbe140f0b70 msgr2=0x7fbe140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:24.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:24.480+0000 7fbe1a031700 1 -- 172.21.15.146:0/4235778613 shutdown_connections 2024-01-21T20:01:24.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:24.480+0000 7fbe1a031700 1 -- 172.21.15.146:0/4235778613 wait complete. 2024-01-21T20:01:24.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:24.480+0000 7fbe1a031700 1 Processor -- start 2024-01-21T20:01:24.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:24.481+0000 7fbe1a031700 1 -- start start 2024-01-21T20:01:24.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:24.481+0000 7fbe1a031700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe140f5410 0x7fbe14109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:24.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:24.481+0000 7fbe1a031700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe14060220 con 0x7fbe140f5410 2024-01-21T20:01:24.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:24.481+0000 7fbe137fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe140f5410 0x7fbe14109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:24.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:24.481+0000 7fbe137fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe140f5410 0x7fbe14109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:54798/0 (socket says 172.21.15.146:54798) 2024-01-21T20:01:24.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:24.481+0000 7fbe137fe700 1 -- 172.21.15.146:0/764246470 learned_addr learned my addr 172.21.15.146:0/764246470 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:01:24.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:24.482+0000 7fbe137fe700 1 -- 172.21.15.146:0/764246470 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe08005040 con 0x7fbe140f5410 2024-01-21T20:01:24.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:24.482+0000 7fbe137fe700 1 --2- 172.21.15.146:0/764246470 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe140f5410 0x7fbe14109450 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fbe08013510 tx=0x7fbe0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:24.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:24.482+0000 7fbdfbfff700 1 -- 172.21.15.146:0/764246470 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe08009320 con 0x7fbe140f5410 2024-01-21T20:01:24.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:24.482+0000 7fbe1a031700 1 -- 172.21.15.146:0/764246470 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe14060420 con 0x7fbe140f5410 2024-01-21T20:01:24.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:24.482+0000 7fbdfbfff700 1 -- 172.21.15.146:0/764246470 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbe08013f00 con 0x7fbe140f5410 2024-01-21T20:01:24.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:24.482+0000 7fbdfbfff700 1 -- 172.21.15.146:0/764246470 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe08014d90 con 0x7fbe140f5410 2024-01-21T20:01:24.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:24.482+0000 7fbe1a031700 1 -- 172.21.15.146:0/764246470 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe14109b00 con 0x7fbe140f5410 2024-01-21T20:01:24.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:24.483+0000 7fbdfbfff700 1 -- 172.21.15.146:0/764246470 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbe080263a0 con 0x7fbe140f5410 2024-01-21T20:01:24.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:24.483+0000 7fbe1a031700 1 -- 172.21.15.146:0/764246470 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe14059100 con 0x7fbe140f5410 2024-01-21T20:01:24.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:24.483+0000 7fbdfbfff700 1 --2- 172.21.15.146:0/764246470 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fbdfc02ff60 0x7fbdfc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:24.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:24.483+0000 7fbdfbfff700 1 -- 172.21.15.146:0/764246470 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbe080413b0 con 0x7fbe140f5410 2024-01-21T20:01:24.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:24.483+0000 7fbe12ffd700 1 --2- 172.21.15.146:0/764246470 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fbdfc02ff60 0x7fbdfc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:24.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:24.484+0000 7fbe12ffd700 1 --2- 172.21.15.146:0/764246470 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fbdfc02ff60 0x7fbdfc032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fbe040096a0 tx=0x7fbe04006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:24.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:24.489+0000 7fbdfbfff700 1 -- 172.21.15.146:0/764246470 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbe080126f0 con 0x7fbe140f5410 2024-01-21T20:01:24.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:24.696+0000 7fbe1a031700 1 -- 172.21.15.146:0/764246470 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbe14045ad0 con 0x7fbe140f5410 2024-01-21T20:01:24.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:24.697+0000 7fbdfbfff700 1 -- 172.21.15.146:0/764246470 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbe08012330 con 0x7fbe140f5410 2024-01-21T20:01:24.687 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:01:24.687 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:01:24.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:24.698+0000 7fbe1a031700 1 -- 172.21.15.146:0/764246470 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fbdfc02ff60 msgr2=0x7fbdfc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:24.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:24.698+0000 7fbe1a031700 1 --2- 172.21.15.146:0/764246470 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fbdfc02ff60 0x7fbdfc032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fbe040096a0 tx=0x7fbe04006b40).stop 2024-01-21T20:01:24.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:24.698+0000 7fbe1a031700 1 -- 172.21.15.146:0/764246470 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe140f5410 msgr2=0x7fbe14109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:24.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:24.698+0000 7fbe1a031700 1 --2- 172.21.15.146:0/764246470 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe140f5410 0x7fbe14109450 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fbe08013510 tx=0x7fbe0800b040).stop 2024-01-21T20:01:24.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:24.698+0000 7fbe1a031700 1 -- 172.21.15.146:0/764246470 shutdown_connections 2024-01-21T20:01:24.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:24.698+0000 7fbe1a031700 1 --2- 172.21.15.146:0/764246470 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fbdfc02ff60 0x7fbdfc032420 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:24.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:24.698+0000 7fbe1a031700 1 --2- 172.21.15.146:0/764246470 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe140f5410 0x7fbe14109450 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:24.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:24.698+0000 7fbe1a031700 1 -- 172.21.15.146:0/764246470 >> 172.21.15.146:0/764246470 conn(0x7fbe140f0b70 msgr2=0x7fbe140f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:24.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:24.699+0000 7fbe1a031700 1 -- 172.21.15.146:0/764246470 shutdown_connections 2024-01-21T20:01:24.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:24.699+0000 7fbe1a031700 1 -- 172.21.15.146:0/764246470 wait complete. 2024-01-21T20:01:24.690 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:01:24.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:24 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:01:23.145322+0000 mgr.smithi053.fgsmol (mgr.14164) 68 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:25.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:25 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:01:24.696999+0000 mon.smithi053 (mon.0) 158 : audit [DBG] from='client.? 172.21.15.146:0/764246470' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:01:26.106 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:01:26.107 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:01:26.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:26 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:01:25.145607+0000 mgr.smithi053.fgsmol (mgr.14164) 69 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:27.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:27.425+0000 7fb36d9d7700 1 -- 172.21.15.146:0/1231806100 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3680f5410 msgr2=0x7fb3680f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:27.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:27.425+0000 7fb36d9d7700 1 --2- 172.21.15.146:0/1231806100 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3680f5410 0x7fb3680f5830 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fb358009fb0 tx=0x7fb358009b30).stop 2024-01-21T20:01:27.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:27.425+0000 7fb36d9d7700 1 -- 172.21.15.146:0/1231806100 shutdown_connections 2024-01-21T20:01:27.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:27.425+0000 7fb36d9d7700 1 --2- 172.21.15.146:0/1231806100 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3680f5410 0x7fb3680f5830 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:27.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:27.425+0000 7fb36d9d7700 1 -- 172.21.15.146:0/1231806100 >> 172.21.15.146:0/1231806100 conn(0x7fb3680f0b70 msgr2=0x7fb3680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:27.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:27.425+0000 7fb36d9d7700 1 -- 172.21.15.146:0/1231806100 shutdown_connections 2024-01-21T20:01:27.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:27.425+0000 7fb36d9d7700 1 -- 172.21.15.146:0/1231806100 wait complete. 2024-01-21T20:01:27.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:27.425+0000 7fb36d9d7700 1 Processor -- start 2024-01-21T20:01:27.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:27.425+0000 7fb36d9d7700 1 -- start start 2024-01-21T20:01:27.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:27.426+0000 7fb36d9d7700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3680f5410 0x7fb368109420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:27.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:27.426+0000 7fb36d9d7700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb368060220 con 0x7fb3680f5410 2024-01-21T20:01:27.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:27.426+0000 7fb366ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3680f5410 0x7fb368109420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:27.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:27.426+0000 7fb366ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3680f5410 0x7fb368109420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:36668/0 (socket says 172.21.15.146:36668) 2024-01-21T20:01:27.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:27.426+0000 7fb366ffd700 1 -- 172.21.15.146:0/1156796596 learned_addr learned my addr 172.21.15.146:0/1156796596 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:01:27.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:27.426+0000 7fb366ffd700 1 -- 172.21.15.146:0/1156796596 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb358005040 con 0x7fb3680f5410 2024-01-21T20:01:27.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:27.426+0000 7fb366ffd700 1 --2- 172.21.15.146:0/1156796596 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3680f5410 0x7fb368109420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fb358013510 tx=0x7fb35800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:27.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:27.426+0000 7fb34f7fe700 1 -- 172.21.15.146:0/1156796596 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb358009320 con 0x7fb3680f5410 2024-01-21T20:01:27.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:27.427+0000 7fb36d9d7700 1 -- 172.21.15.146:0/1156796596 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb368060420 con 0x7fb3680f5410 2024-01-21T20:01:27.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:27.427+0000 7fb34f7fe700 1 -- 172.21.15.146:0/1156796596 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb358013f00 con 0x7fb3680f5410 2024-01-21T20:01:27.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:27.427+0000 7fb34f7fe700 1 -- 172.21.15.146:0/1156796596 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb358014d90 con 0x7fb3680f5410 2024-01-21T20:01:27.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:27.427+0000 7fb36d9d7700 1 -- 172.21.15.146:0/1156796596 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb368109ad0 con 0x7fb3680f5410 2024-01-21T20:01:27.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:27.427+0000 7fb34f7fe700 1 -- 172.21.15.146:0/1156796596 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb3580263a0 con 0x7fb3680f5410 2024-01-21T20:01:27.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:27.427+0000 7fb34f7fe700 1 --2- 172.21.15.146:0/1156796596 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb35002ff60 0x7fb350032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:27.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:27.428+0000 7fb34f7fe700 1 -- 172.21.15.146:0/1156796596 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb3580410e0 con 0x7fb3680f5410 2024-01-21T20:01:27.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:27.428+0000 7fb36d9d7700 1 -- 172.21.15.146:0/1156796596 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb368059100 con 0x7fb3680f5410 2024-01-21T20:01:27.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:27.428+0000 7fb3667fc700 1 --2- 172.21.15.146:0/1156796596 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb35002ff60 0x7fb350032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:27.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:27.428+0000 7fb3667fc700 1 --2- 172.21.15.146:0/1156796596 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb35002ff60 0x7fb350032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fb35c0096a0 tx=0x7fb35c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:27.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:27.433+0000 7fb34f7fe700 1 -- 172.21.15.146:0/1156796596 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb358012920 con 0x7fb3680f5410 2024-01-21T20:01:27.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:27.635+0000 7fb36d9d7700 1 -- 172.21.15.146:0/1156796596 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb368045ad0 con 0x7fb3680f5410 2024-01-21T20:01:27.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:27.636+0000 7fb34f7fe700 1 -- 172.21.15.146:0/1156796596 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb35802b020 con 0x7fb3680f5410 2024-01-21T20:01:27.627 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:01:27.627 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:01:27.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:27.638+0000 7fb36d9d7700 1 -- 172.21.15.146:0/1156796596 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb35002ff60 msgr2=0x7fb350032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:27.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:27.638+0000 7fb36d9d7700 1 --2- 172.21.15.146:0/1156796596 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb35002ff60 0x7fb350032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fb35c0096a0 tx=0x7fb35c006b40).stop 2024-01-21T20:01:27.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:27.638+0000 7fb36d9d7700 1 -- 172.21.15.146:0/1156796596 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3680f5410 msgr2=0x7fb368109420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:27.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:27.638+0000 7fb36d9d7700 1 --2- 172.21.15.146:0/1156796596 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3680f5410 0x7fb368109420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fb358013510 tx=0x7fb35800b040).stop 2024-01-21T20:01:27.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:27.638+0000 7fb36d9d7700 1 -- 172.21.15.146:0/1156796596 shutdown_connections 2024-01-21T20:01:27.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:27.638+0000 7fb36d9d7700 1 --2- 172.21.15.146:0/1156796596 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb35002ff60 0x7fb350032420 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:27.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:27.638+0000 7fb36d9d7700 1 --2- 172.21.15.146:0/1156796596 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3680f5410 0x7fb368109420 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:27.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:27.638+0000 7fb36d9d7700 1 -- 172.21.15.146:0/1156796596 >> 172.21.15.146:0/1156796596 conn(0x7fb3680f0b70 msgr2=0x7fb3680f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:27.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:27.638+0000 7fb36d9d7700 1 -- 172.21.15.146:0/1156796596 shutdown_connections 2024-01-21T20:01:27.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:27.638+0000 7fb36d9d7700 1 -- 172.21.15.146:0/1156796596 wait complete. 2024-01-21T20:01:27.629 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:01:28.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:28 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:01:27.145978+0000 mgr.smithi053.fgsmol (mgr.14164) 70 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:28.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:28 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:01:27.636446+0000 mon.smithi053 (mon.0) 159 : audit [DBG] from='client.? 172.21.15.146:0/1156796596' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:01:29.032 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:01:29.032 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:01:30.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:30.267+0000 7f4c5f82f700 1 -- 172.21.15.146:0/3198864150 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c580f6510 msgr2=0x7f4c580f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:30.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:30.267+0000 7f4c5f82f700 1 --2- 172.21.15.146:0/3198864150 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c580f6510 0x7f4c580f6930 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f4c4c009fb0 tx=0x7f4c4c009b30).stop 2024-01-21T20:01:30.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:30.267+0000 7f4c5f82f700 1 -- 172.21.15.146:0/3198864150 shutdown_connections 2024-01-21T20:01:30.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:30.267+0000 7f4c5f82f700 1 --2- 172.21.15.146:0/3198864150 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c580f6510 0x7f4c580f6930 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:30.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:30.267+0000 7f4c5f82f700 1 -- 172.21.15.146:0/3198864150 >> 172.21.15.146:0/3198864150 conn(0x7f4c580f1c70 msgr2=0x7f4c580f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:30.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:30.267+0000 7f4c5f82f700 1 -- 172.21.15.146:0/3198864150 shutdown_connections 2024-01-21T20:01:30.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:30.267+0000 7f4c5f82f700 1 -- 172.21.15.146:0/3198864150 wait complete. 2024-01-21T20:01:30.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:30.268+0000 7f4c5f82f700 1 Processor -- start 2024-01-21T20:01:30.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:30.268+0000 7f4c5f82f700 1 -- start start 2024-01-21T20:01:30.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:30.268+0000 7f4c5f82f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c580f6510 0x7f4c58106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:30.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:30.268+0000 7f4c5f82f700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c58106c20 con 0x7f4c580f6510 2024-01-21T20:01:30.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:30.268+0000 7f4c5d5cb700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c580f6510 0x7f4c58106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:30.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:30.268+0000 7f4c5d5cb700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c580f6510 0x7f4c58106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:36692/0 (socket says 172.21.15.146:36692) 2024-01-21T20:01:30.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:30.268+0000 7f4c5d5cb700 1 -- 172.21.15.146:0/922674616 learned_addr learned my addr 172.21.15.146:0/922674616 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:01:30.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:30.269+0000 7f4c5d5cb700 1 -- 172.21.15.146:0/922674616 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c4c005040 con 0x7f4c580f6510 2024-01-21T20:01:30.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:30.269+0000 7f4c5d5cb700 1 --2- 172.21.15.146:0/922674616 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c580f6510 0x7f4c58106730 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f4c4c013510 tx=0x7f4c4c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:30.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:30.269+0000 7f4c49ffb700 1 -- 172.21.15.146:0/922674616 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c4c009320 con 0x7f4c580f6510 2024-01-21T20:01:30.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:30.269+0000 7f4c5f82f700 1 -- 172.21.15.146:0/922674616 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c58106e20 con 0x7f4c580f6510 2024-01-21T20:01:30.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:30.269+0000 7f4c49ffb700 1 -- 172.21.15.146:0/922674616 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4c4c013f00 con 0x7f4c580f6510 2024-01-21T20:01:30.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:30.269+0000 7f4c49ffb700 1 -- 172.21.15.146:0/922674616 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c4c014d90 con 0x7f4c580f6510 2024-01-21T20:01:30.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:30.269+0000 7f4c5f82f700 1 -- 172.21.15.146:0/922674616 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c581072b0 con 0x7f4c580f6510 2024-01-21T20:01:30.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:30.270+0000 7f4c49ffb700 1 -- 172.21.15.146:0/922674616 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4c4c0095f0 con 0x7f4c580f6510 2024-01-21T20:01:30.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:30.270+0000 7f4c5f82f700 1 -- 172.21.15.146:0/922674616 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c58045ad0 con 0x7f4c580f6510 2024-01-21T20:01:30.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:30.270+0000 7f4c49ffb700 1 --2- 172.21.15.146:0/922674616 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f4c4402ffb0 0x7f4c44032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:30.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:30.270+0000 7f4c49ffb700 1 -- 172.21.15.146:0/922674616 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4c4c040f80 con 0x7f4c580f6510 2024-01-21T20:01:30.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:30.270+0000 7f4c5cdca700 1 --2- 172.21.15.146:0/922674616 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f4c4402ffb0 0x7f4c44032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:30.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:30.271+0000 7f4c5cdca700 1 --2- 172.21.15.146:0/922674616 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f4c4402ffb0 0x7f4c44032470 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f4c540096a0 tx=0x7f4c54006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:30.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:30.275+0000 7f4c49ffb700 1 -- 172.21.15.146:0/922674616 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4c4c014420 con 0x7f4c580f6510 2024-01-21T20:01:30.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:30.477+0000 7f4c5f82f700 1 -- 172.21.15.146:0/922674616 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4c580fbf60 con 0x7f4c580f6510 2024-01-21T20:01:30.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:30.478+0000 7f4c49ffb700 1 -- 172.21.15.146:0/922674616 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4c4c01d7f0 con 0x7f4c580f6510 2024-01-21T20:01:30.471 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:01:30.471 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:01:30.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:30.479+0000 7f4c5f82f700 1 -- 172.21.15.146:0/922674616 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f4c4402ffb0 msgr2=0x7f4c44032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:30.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:30.479+0000 7f4c5f82f700 1 --2- 172.21.15.146:0/922674616 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f4c4402ffb0 0x7f4c44032470 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f4c540096a0 tx=0x7f4c54006b40).stop 2024-01-21T20:01:30.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:30.479+0000 7f4c5f82f700 1 -- 172.21.15.146:0/922674616 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c580f6510 msgr2=0x7f4c58106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:30.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:30.479+0000 7f4c5f82f700 1 --2- 172.21.15.146:0/922674616 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c580f6510 0x7f4c58106730 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f4c4c013510 tx=0x7f4c4c00b040).stop 2024-01-21T20:01:30.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:30.479+0000 7f4c5f82f700 1 -- 172.21.15.146:0/922674616 shutdown_connections 2024-01-21T20:01:30.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:30.479+0000 7f4c5f82f700 1 --2- 172.21.15.146:0/922674616 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f4c4402ffb0 0x7f4c44032470 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:30.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:30.479+0000 7f4c5f82f700 1 --2- 172.21.15.146:0/922674616 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c580f6510 0x7f4c58106730 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:30.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:30.479+0000 7f4c5f82f700 1 -- 172.21.15.146:0/922674616 >> 172.21.15.146:0/922674616 conn(0x7f4c580f1c70 msgr2=0x7f4c580f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:30.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:30.479+0000 7f4c5f82f700 1 -- 172.21.15.146:0/922674616 shutdown_connections 2024-01-21T20:01:30.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:30.479+0000 7f4c5f82f700 1 -- 172.21.15.146:0/922674616 wait complete. 2024-01-21T20:01:30.473 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:01:30.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:30 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:01:29.146312+0000 mgr.smithi053.fgsmol (mgr.14164) 71 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:30.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:30 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:01:30.478078+0000 mon.smithi053 (mon.0) 160 : audit [DBG] from='client.? 172.21.15.146:0/922674616' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:01:31.870 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:01:31.870 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:01:32.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:32 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:01:31.146690+0000 mgr.smithi053.fgsmol (mgr.14164) 72 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:33.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:33.117+0000 7f8462ca2700 1 -- 172.21.15.146:0/4214368594 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f845c0f5410 msgr2=0x7f845c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:33.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:33.117+0000 7f8462ca2700 1 --2- 172.21.15.146:0/4214368594 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f845c0f5410 0x7f845c0f5830 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f8450009fb0 tx=0x7f8450009b30).stop 2024-01-21T20:01:33.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:33.117+0000 7f8462ca2700 1 -- 172.21.15.146:0/4214368594 shutdown_connections 2024-01-21T20:01:33.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:33.117+0000 7f8462ca2700 1 --2- 172.21.15.146:0/4214368594 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f845c0f5410 0x7f845c0f5830 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:33.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:33.117+0000 7f8462ca2700 1 -- 172.21.15.146:0/4214368594 >> 172.21.15.146:0/4214368594 conn(0x7f845c0f0b70 msgr2=0x7f845c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:33.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:33.117+0000 7f8462ca2700 1 -- 172.21.15.146:0/4214368594 shutdown_connections 2024-01-21T20:01:33.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:33.117+0000 7f8462ca2700 1 -- 172.21.15.146:0/4214368594 wait complete. 2024-01-21T20:01:33.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:33.118+0000 7f8462ca2700 1 Processor -- start 2024-01-21T20:01:33.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:33.118+0000 7f8462ca2700 1 -- start start 2024-01-21T20:01:33.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:33.118+0000 7f8462ca2700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f845c0f5410 0x7f845c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:33.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:33.118+0000 7f8462ca2700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f845c106c20 con 0x7f845c0f5410 2024-01-21T20:01:33.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:33.118+0000 7f8460a3e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f845c0f5410 0x7f845c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:33.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:33.118+0000 7f8460a3e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f845c0f5410 0x7f845c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:36710/0 (socket says 172.21.15.146:36710) 2024-01-21T20:01:33.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:33.118+0000 7f8460a3e700 1 -- 172.21.15.146:0/2428629742 learned_addr learned my addr 172.21.15.146:0/2428629742 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:01:33.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:33.119+0000 7f8460a3e700 1 -- 172.21.15.146:0/2428629742 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8450005040 con 0x7f845c0f5410 2024-01-21T20:01:33.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:33.119+0000 7f8460a3e700 1 --2- 172.21.15.146:0/2428629742 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f845c0f5410 0x7f845c106730 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f8450013510 tx=0x7f845000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:33.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:33.119+0000 7f84597fa700 1 -- 172.21.15.146:0/2428629742 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8450009320 con 0x7f845c0f5410 2024-01-21T20:01:33.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:33.119+0000 7f8462ca2700 1 -- 172.21.15.146:0/2428629742 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f845c106e20 con 0x7f845c0f5410 2024-01-21T20:01:33.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:33.119+0000 7f84597fa700 1 -- 172.21.15.146:0/2428629742 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8450013f00 con 0x7f845c0f5410 2024-01-21T20:01:33.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:33.119+0000 7f84597fa700 1 -- 172.21.15.146:0/2428629742 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8450014d90 con 0x7f845c0f5410 2024-01-21T20:01:33.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:33.119+0000 7f8462ca2700 1 -- 172.21.15.146:0/2428629742 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f845c1072b0 con 0x7f845c0f5410 2024-01-21T20:01:33.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:33.120+0000 7f84597fa700 1 -- 172.21.15.146:0/2428629742 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f84500095f0 con 0x7f845c0f5410 2024-01-21T20:01:33.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:33.120+0000 7f8462ca2700 1 -- 172.21.15.146:0/2428629742 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f845c045ad0 con 0x7f845c0f5410 2024-01-21T20:01:33.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:33.120+0000 7f84597fa700 1 --2- 172.21.15.146:0/2428629742 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f844c02ff60 0x7f844c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:33.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:33.120+0000 7f84597fa700 1 -- 172.21.15.146:0/2428629742 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8450040fc0 con 0x7f845c0f5410 2024-01-21T20:01:33.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:33.120+0000 7f845bfff700 1 --2- 172.21.15.146:0/2428629742 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f844c02ff60 0x7f844c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:33.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:33.121+0000 7f845bfff700 1 --2- 172.21.15.146:0/2428629742 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f844c02ff60 0x7f844c032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f84480096a0 tx=0x7f8448006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:33.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:33.125+0000 7f84597fa700 1 -- 172.21.15.146:0/2428629742 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f845006d2b0 con 0x7f845c0f5410 2024-01-21T20:01:33.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:33.330+0000 7f8462ca2700 1 -- 172.21.15.146:0/2428629742 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f845c0fae60 con 0x7f845c0f5410 2024-01-21T20:01:33.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:33.331+0000 7f84597fa700 1 -- 172.21.15.146:0/2428629742 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f84500263a0 con 0x7f845c0f5410 2024-01-21T20:01:33.322 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:01:33.322 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:01:33.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:33.333+0000 7f8462ca2700 1 -- 172.21.15.146:0/2428629742 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f844c02ff60 msgr2=0x7f844c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:33.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:33.333+0000 7f8462ca2700 1 --2- 172.21.15.146:0/2428629742 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f844c02ff60 0x7f844c032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f84480096a0 tx=0x7f8448006b40).stop 2024-01-21T20:01:33.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:33.333+0000 7f8462ca2700 1 -- 172.21.15.146:0/2428629742 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f845c0f5410 msgr2=0x7f845c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:33.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:33.333+0000 7f8462ca2700 1 --2- 172.21.15.146:0/2428629742 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f845c0f5410 0x7f845c106730 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f8450013510 tx=0x7f845000b040).stop 2024-01-21T20:01:33.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:33.333+0000 7f8462ca2700 1 -- 172.21.15.146:0/2428629742 shutdown_connections 2024-01-21T20:01:33.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:33.333+0000 7f8462ca2700 1 --2- 172.21.15.146:0/2428629742 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f844c02ff60 0x7f844c032420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:33.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:33.333+0000 7f8462ca2700 1 --2- 172.21.15.146:0/2428629742 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f845c0f5410 0x7f845c106730 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:33.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:33.333+0000 7f8462ca2700 1 -- 172.21.15.146:0/2428629742 >> 172.21.15.146:0/2428629742 conn(0x7f845c0f0b70 msgr2=0x7f845c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:33.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:33.333+0000 7f8462ca2700 1 -- 172.21.15.146:0/2428629742 shutdown_connections 2024-01-21T20:01:33.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:33.333+0000 7f8462ca2700 1 -- 172.21.15.146:0/2428629742 wait complete. 2024-01-21T20:01:33.324 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:01:33.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:33 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:01:33.331535+0000 mon.smithi053 (mon.0) 161 : audit [DBG] from='client.? 172.21.15.146:0/2428629742' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:01:34.719 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:01:34.720 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:01:34.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:34 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:01:33.147027+0000 mgr.smithi053.fgsmol (mgr.14164) 73 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:35.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:35.966+0000 7ff5ab701700 1 -- 172.21.15.146:0/1513780234 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff5a40f5410 msgr2=0x7ff5a40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:35.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:35.966+0000 7ff5ab701700 1 --2- 172.21.15.146:0/1513780234 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff5a40f5410 0x7ff5a40f5830 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7ff594009fb0 tx=0x7ff594009b30).stop 2024-01-21T20:01:35.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:35.966+0000 7ff5ab701700 1 -- 172.21.15.146:0/1513780234 shutdown_connections 2024-01-21T20:01:35.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:35.966+0000 7ff5ab701700 1 --2- 172.21.15.146:0/1513780234 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff5a40f5410 0x7ff5a40f5830 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:35.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:35.966+0000 7ff5ab701700 1 -- 172.21.15.146:0/1513780234 >> 172.21.15.146:0/1513780234 conn(0x7ff5a40f0b70 msgr2=0x7ff5a40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:35.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:35.967+0000 7ff5ab701700 1 -- 172.21.15.146:0/1513780234 shutdown_connections 2024-01-21T20:01:35.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:35.967+0000 7ff5ab701700 1 -- 172.21.15.146:0/1513780234 wait complete. 2024-01-21T20:01:35.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:35.967+0000 7ff5ab701700 1 Processor -- start 2024-01-21T20:01:35.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:35.967+0000 7ff5ab701700 1 -- start start 2024-01-21T20:01:35.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:35.967+0000 7ff5ab701700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff5a40f5410 0x7ff5a4109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:35.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:35.967+0000 7ff5ab701700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a4060240 con 0x7ff5a40f5410 2024-01-21T20:01:35.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:35.967+0000 7ff5a949d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff5a40f5410 0x7ff5a4109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:35.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:35.967+0000 7ff5a949d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff5a40f5410 0x7ff5a4109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:36718/0 (socket says 172.21.15.146:36718) 2024-01-21T20:01:35.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:35.967+0000 7ff5a949d700 1 -- 172.21.15.146:0/2688204150 learned_addr learned my addr 172.21.15.146:0/2688204150 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:01:35.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:35.968+0000 7ff5a949d700 1 -- 172.21.15.146:0/2688204150 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff594005040 con 0x7ff5a40f5410 2024-01-21T20:01:35.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:35.968+0000 7ff5a949d700 1 --2- 172.21.15.146:0/2688204150 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff5a40f5410 0x7ff5a4109430 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7ff594013510 tx=0x7ff59400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:35.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:35.968+0000 7ff599ffb700 1 -- 172.21.15.146:0/2688204150 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff594009320 con 0x7ff5a40f5410 2024-01-21T20:01:35.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:35.968+0000 7ff5ab701700 1 -- 172.21.15.146:0/2688204150 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5a4060440 con 0x7ff5a40f5410 2024-01-21T20:01:35.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:35.968+0000 7ff599ffb700 1 -- 172.21.15.146:0/2688204150 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff594013f00 con 0x7ff5a40f5410 2024-01-21T20:01:35.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:35.968+0000 7ff599ffb700 1 -- 172.21.15.146:0/2688204150 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff594014d90 con 0x7ff5a40f5410 2024-01-21T20:01:35.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:35.968+0000 7ff5ab701700 1 -- 172.21.15.146:0/2688204150 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5a4109ae0 con 0x7ff5a40f5410 2024-01-21T20:01:35.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:35.969+0000 7ff599ffb700 1 -- 172.21.15.146:0/2688204150 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff5940095f0 con 0x7ff5a40f5410 2024-01-21T20:01:35.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:35.969+0000 7ff5ab701700 1 -- 172.21.15.146:0/2688204150 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5a4045ad0 con 0x7ff5a40f5410 2024-01-21T20:01:35.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:35.969+0000 7ff599ffb700 1 --2- 172.21.15.146:0/2688204150 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff59002ff60 0x7ff590032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:35.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:35.969+0000 7ff599ffb700 1 -- 172.21.15.146:0/2688204150 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff594040fe0 con 0x7ff5a40f5410 2024-01-21T20:01:35.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:35.969+0000 7ff5a8c9c700 1 --2- 172.21.15.146:0/2688204150 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff59002ff60 0x7ff590032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:35.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:35.970+0000 7ff5a8c9c700 1 --2- 172.21.15.146:0/2688204150 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff59002ff60 0x7ff590032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7ff5a00096a0 tx=0x7ff5a0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:35.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:35.974+0000 7ff599ffb700 1 -- 172.21.15.146:0/2688204150 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff594014420 con 0x7ff5a40f5410 2024-01-21T20:01:36.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:36.175+0000 7ff5ab701700 1 -- 172.21.15.146:0/2688204150 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff5a40fae60 con 0x7ff5a40f5410 2024-01-21T20:01:36.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:36.176+0000 7ff599ffb700 1 -- 172.21.15.146:0/2688204150 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff594010230 con 0x7ff5a40f5410 2024-01-21T20:01:36.166 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:01:36.166 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:01:36.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:36.177+0000 7ff5ab701700 1 -- 172.21.15.146:0/2688204150 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff59002ff60 msgr2=0x7ff590032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:36.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:36.177+0000 7ff5ab701700 1 --2- 172.21.15.146:0/2688204150 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff59002ff60 0x7ff590032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7ff5a00096a0 tx=0x7ff5a0006b40).stop 2024-01-21T20:01:36.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:36.177+0000 7ff5ab701700 1 -- 172.21.15.146:0/2688204150 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff5a40f5410 msgr2=0x7ff5a4109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:36.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:36.177+0000 7ff5ab701700 1 --2- 172.21.15.146:0/2688204150 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff5a40f5410 0x7ff5a4109430 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7ff594013510 tx=0x7ff59400b040).stop 2024-01-21T20:01:36.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:36.177+0000 7ff5ab701700 1 -- 172.21.15.146:0/2688204150 shutdown_connections 2024-01-21T20:01:36.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:36.177+0000 7ff5ab701700 1 --2- 172.21.15.146:0/2688204150 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff59002ff60 0x7ff590032420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:36.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:36.177+0000 7ff5ab701700 1 --2- 172.21.15.146:0/2688204150 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff5a40f5410 0x7ff5a4109430 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:36.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:36.177+0000 7ff5ab701700 1 -- 172.21.15.146:0/2688204150 >> 172.21.15.146:0/2688204150 conn(0x7ff5a40f0b70 msgr2=0x7ff5a40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:36.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:36.178+0000 7ff5ab701700 1 -- 172.21.15.146:0/2688204150 shutdown_connections 2024-01-21T20:01:36.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:36.178+0000 7ff5ab701700 1 -- 172.21.15.146:0/2688204150 wait complete. 2024-01-21T20:01:36.169 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:01:36.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:36 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:01:35.147418+0000 mgr.smithi053.fgsmol (mgr.14164) 74 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:36.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:36 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:01:36.176209+0000 mon.smithi053 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.146:0/2688204150' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:01:37.582 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:01:37.582 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:01:38.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:38.823+0000 7ff9a32b0700 1 -- 172.21.15.146:0/4050792585 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff99c0f5410 msgr2=0x7ff99c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:38.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:38.823+0000 7ff9a32b0700 1 --2- 172.21.15.146:0/4050792585 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff99c0f5410 0x7ff99c0f5830 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7ff98c009fb0 tx=0x7ff98c009b30).stop 2024-01-21T20:01:38.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:38.823+0000 7ff9a32b0700 1 -- 172.21.15.146:0/4050792585 shutdown_connections 2024-01-21T20:01:38.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:38.823+0000 7ff9a32b0700 1 --2- 172.21.15.146:0/4050792585 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff99c0f5410 0x7ff99c0f5830 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:38.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:38.823+0000 7ff9a32b0700 1 -- 172.21.15.146:0/4050792585 >> 172.21.15.146:0/4050792585 conn(0x7ff99c0f0b70 msgr2=0x7ff99c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:38.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:38.823+0000 7ff9a32b0700 1 -- 172.21.15.146:0/4050792585 shutdown_connections 2024-01-21T20:01:38.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:38.823+0000 7ff9a32b0700 1 -- 172.21.15.146:0/4050792585 wait complete. 2024-01-21T20:01:38.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:38.824+0000 7ff9a32b0700 1 Processor -- start 2024-01-21T20:01:38.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:38.824+0000 7ff9a32b0700 1 -- start start 2024-01-21T20:01:38.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:38.824+0000 7ff9a32b0700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff99c060240 0x7ff99c109520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:38.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:38.824+0000 7ff9a32b0700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff99c109a10 con 0x7ff99c060240 2024-01-21T20:01:38.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:38.824+0000 7ff9a104c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff99c060240 0x7ff99c109520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:38.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:38.824+0000 7ff9a104c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff99c060240 0x7ff99c109520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:33016/0 (socket says 172.21.15.146:33016) 2024-01-21T20:01:38.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:38.824+0000 7ff9a104c700 1 -- 172.21.15.146:0/2816972912 learned_addr learned my addr 172.21.15.146:0/2816972912 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:01:38.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:38.825+0000 7ff9a104c700 1 -- 172.21.15.146:0/2816972912 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff98c005040 con 0x7ff99c060240 2024-01-21T20:01:38.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:38.825+0000 7ff9a104c700 1 --2- 172.21.15.146:0/2816972912 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff99c060240 0x7ff99c109520 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7ff98c009f80 tx=0x7ff98c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:38.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:38.825+0000 7ff991ffb700 1 -- 172.21.15.146:0/2816972912 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff98c0136c0 con 0x7ff99c060240 2024-01-21T20:01:38.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:38.825+0000 7ff9a32b0700 1 -- 172.21.15.146:0/2816972912 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff99c109b50 con 0x7ff99c060240 2024-01-21T20:01:38.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:38.825+0000 7ff991ffb700 1 -- 172.21.15.146:0/2816972912 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff98c0095f0 con 0x7ff99c060240 2024-01-21T20:01:38.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:38.825+0000 7ff991ffb700 1 -- 172.21.15.146:0/2816972912 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff98c014420 con 0x7ff99c060240 2024-01-21T20:01:38.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:38.825+0000 7ff9a32b0700 1 -- 172.21.15.146:0/2816972912 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff99c109fe0 con 0x7ff99c060240 2024-01-21T20:01:38.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:38.826+0000 7ff991ffb700 1 -- 172.21.15.146:0/2816972912 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff98c025060 con 0x7ff99c060240 2024-01-21T20:01:38.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:38.826+0000 7ff9a32b0700 1 -- 172.21.15.146:0/2816972912 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff99c059100 con 0x7ff99c060240 2024-01-21T20:01:38.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:38.826+0000 7ff991ffb700 1 --2- 172.21.15.146:0/2816972912 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff98802ff60 0x7ff988032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:38.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:38.826+0000 7ff991ffb700 1 -- 172.21.15.146:0/2816972912 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff98c040d80 con 0x7ff99c060240 2024-01-21T20:01:38.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:38.826+0000 7ff9a084b700 1 --2- 172.21.15.146:0/2816972912 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff98802ff60 0x7ff988032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:38.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:38.827+0000 7ff9a084b700 1 --2- 172.21.15.146:0/2816972912 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff98802ff60 0x7ff988032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7ff9980096a0 tx=0x7ff998006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:38.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:38.831+0000 7ff991ffb700 1 -- 172.21.15.146:0/2816972912 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff98c01cb60 con 0x7ff99c060240 2024-01-21T20:01:38.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:38 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:01:37.147680+0000 mgr.smithi053.fgsmol (mgr.14164) 75 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:39.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:39.037+0000 7ff9a32b0700 1 -- 172.21.15.146:0/2816972912 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff99c045ad0 con 0x7ff99c060240 2024-01-21T20:01:39.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:39.038+0000 7ff991ffb700 1 -- 172.21.15.146:0/2816972912 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff99c045ad0 con 0x7ff99c060240 2024-01-21T20:01:39.029 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:01:39.029 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:01:39.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:39.039+0000 7ff9a32b0700 1 -- 172.21.15.146:0/2816972912 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff98802ff60 msgr2=0x7ff988032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:39.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:39.039+0000 7ff9a32b0700 1 --2- 172.21.15.146:0/2816972912 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff98802ff60 0x7ff988032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7ff9980096a0 tx=0x7ff998006b40).stop 2024-01-21T20:01:39.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:39.039+0000 7ff9a32b0700 1 -- 172.21.15.146:0/2816972912 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff99c060240 msgr2=0x7ff99c109520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:39.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:39.039+0000 7ff9a32b0700 1 --2- 172.21.15.146:0/2816972912 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff99c060240 0x7ff99c109520 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7ff98c009f80 tx=0x7ff98c009b30).stop 2024-01-21T20:01:39.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:39.040+0000 7ff9a32b0700 1 -- 172.21.15.146:0/2816972912 shutdown_connections 2024-01-21T20:01:39.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:39.040+0000 7ff9a32b0700 1 --2- 172.21.15.146:0/2816972912 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff98802ff60 0x7ff988032420 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:39.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:39.040+0000 7ff9a32b0700 1 --2- 172.21.15.146:0/2816972912 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff99c060240 0x7ff99c109520 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:39.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:39.040+0000 7ff9a32b0700 1 -- 172.21.15.146:0/2816972912 >> 172.21.15.146:0/2816972912 conn(0x7ff99c0f0b70 msgr2=0x7ff99c0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:39.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:39.040+0000 7ff9a32b0700 1 -- 172.21.15.146:0/2816972912 shutdown_connections 2024-01-21T20:01:39.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:39.040+0000 7ff9a32b0700 1 -- 172.21.15.146:0/2816972912 wait complete. 2024-01-21T20:01:39.032 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:01:39.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:39 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:01:39.039011+0000 mon.smithi053 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.146:0/2816972912' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:01:40.436 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:01:40.436 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:01:40.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:40 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:01:39.147931+0000 mgr.smithi053.fgsmol (mgr.14164) 76 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:41.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:41.716+0000 7f63c9d3c700 1 -- 172.21.15.146:0/1343761097 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63c40f6510 msgr2=0x7f63c40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:41.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:41.716+0000 7f63c9d3c700 1 --2- 172.21.15.146:0/1343761097 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63c40f6510 0x7f63c40f6930 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f63b8009fb0 tx=0x7f63b8009b30).stop 2024-01-21T20:01:41.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:41.716+0000 7f63c9d3c700 1 -- 172.21.15.146:0/1343761097 shutdown_connections 2024-01-21T20:01:41.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:41.716+0000 7f63c9d3c700 1 --2- 172.21.15.146:0/1343761097 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63c40f6510 0x7f63c40f6930 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:41.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:41.716+0000 7f63c9d3c700 1 -- 172.21.15.146:0/1343761097 >> 172.21.15.146:0/1343761097 conn(0x7f63c40f1c70 msgr2=0x7f63c40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:41.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:41.716+0000 7f63c9d3c700 1 -- 172.21.15.146:0/1343761097 shutdown_connections 2024-01-21T20:01:41.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:41.716+0000 7f63c9d3c700 1 -- 172.21.15.146:0/1343761097 wait complete. 2024-01-21T20:01:41.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:41.717+0000 7f63c9d3c700 1 Processor -- start 2024-01-21T20:01:41.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:41.717+0000 7f63c9d3c700 1 -- start start 2024-01-21T20:01:41.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:41.717+0000 7f63c9d3c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63c40f6510 0x7f63c4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:41.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:41.717+0000 7f63c9d3c700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c4106c20 con 0x7f63c40f6510 2024-01-21T20:01:41.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:41.717+0000 7f63c37fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63c40f6510 0x7f63c4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:41.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:41.717+0000 7f63c37fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63c40f6510 0x7f63c4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:33032/0 (socket says 172.21.15.146:33032) 2024-01-21T20:01:41.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:41.717+0000 7f63c37fe700 1 -- 172.21.15.146:0/2305026890 learned_addr learned my addr 172.21.15.146:0/2305026890 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:01:41.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:41.718+0000 7f63c37fe700 1 -- 172.21.15.146:0/2305026890 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b8005040 con 0x7f63c40f6510 2024-01-21T20:01:41.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:41.718+0000 7f63c37fe700 1 --2- 172.21.15.146:0/2305026890 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63c40f6510 0x7f63c4106730 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f63b8013510 tx=0x7f63b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:41.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:41.718+0000 7f63abfff700 1 -- 172.21.15.146:0/2305026890 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f63b8009320 con 0x7f63c40f6510 2024-01-21T20:01:41.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:41.718+0000 7f63c9d3c700 1 -- 172.21.15.146:0/2305026890 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63c4106e20 con 0x7f63c40f6510 2024-01-21T20:01:41.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:41.718+0000 7f63abfff700 1 -- 172.21.15.146:0/2305026890 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f63b8013f00 con 0x7f63c40f6510 2024-01-21T20:01:41.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:41.718+0000 7f63abfff700 1 -- 172.21.15.146:0/2305026890 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f63b8014d90 con 0x7f63c40f6510 2024-01-21T20:01:41.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:41.718+0000 7f63c9d3c700 1 -- 172.21.15.146:0/2305026890 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63c41072b0 con 0x7f63c40f6510 2024-01-21T20:01:41.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:41.719+0000 7f63abfff700 1 -- 172.21.15.146:0/2305026890 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f63b80095f0 con 0x7f63c40f6510 2024-01-21T20:01:41.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:41.719+0000 7f63c9d3c700 1 -- 172.21.15.146:0/2305026890 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63c4045ad0 con 0x7f63c40f6510 2024-01-21T20:01:41.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:41.719+0000 7f63abfff700 1 --2- 172.21.15.146:0/2305026890 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f63ac02ff60 0x7f63ac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:41.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:41.719+0000 7f63abfff700 1 -- 172.21.15.146:0/2305026890 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f63b8041000 con 0x7f63c40f6510 2024-01-21T20:01:41.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:41.719+0000 7f63c2ffd700 1 --2- 172.21.15.146:0/2305026890 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f63ac02ff60 0x7f63ac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:41.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:41.720+0000 7f63c2ffd700 1 --2- 172.21.15.146:0/2305026890 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f63ac02ff60 0x7f63ac032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f63b40096a0 tx=0x7f63b4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:41.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:41.724+0000 7f63abfff700 1 -- 172.21.15.146:0/2305026890 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f63b8014420 con 0x7f63c40f6510 2024-01-21T20:01:41.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:41.929+0000 7f63c9d3c700 1 -- 172.21.15.146:0/2305026890 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f63c40fbf60 con 0x7f63c40f6510 2024-01-21T20:01:41.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:41.929+0000 7f63abfff700 1 -- 172.21.15.146:0/2305026890 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f63b8010230 con 0x7f63c40f6510 2024-01-21T20:01:41.921 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:01:41.921 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:01:41.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:41.931+0000 7f63c9d3c700 1 -- 172.21.15.146:0/2305026890 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f63ac02ff60 msgr2=0x7f63ac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:41.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:41.931+0000 7f63c9d3c700 1 --2- 172.21.15.146:0/2305026890 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f63ac02ff60 0x7f63ac032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f63b40096a0 tx=0x7f63b4006b40).stop 2024-01-21T20:01:41.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:41.931+0000 7f63c9d3c700 1 -- 172.21.15.146:0/2305026890 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63c40f6510 msgr2=0x7f63c4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:41.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:41.931+0000 7f63c9d3c700 1 --2- 172.21.15.146:0/2305026890 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63c40f6510 0x7f63c4106730 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f63b8013510 tx=0x7f63b800b040).stop 2024-01-21T20:01:41.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:41.931+0000 7f63c9d3c700 1 -- 172.21.15.146:0/2305026890 shutdown_connections 2024-01-21T20:01:41.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:41.931+0000 7f63c9d3c700 1 --2- 172.21.15.146:0/2305026890 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f63ac02ff60 0x7f63ac032420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:41.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:41.931+0000 7f63c9d3c700 1 --2- 172.21.15.146:0/2305026890 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63c40f6510 0x7f63c4106730 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:41.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:41.931+0000 7f63c9d3c700 1 -- 172.21.15.146:0/2305026890 >> 172.21.15.146:0/2305026890 conn(0x7f63c40f1c70 msgr2=0x7f63c40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:41.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:41.931+0000 7f63c9d3c700 1 -- 172.21.15.146:0/2305026890 shutdown_connections 2024-01-21T20:01:41.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:41.931+0000 7f63c9d3c700 1 -- 172.21.15.146:0/2305026890 wait complete. 2024-01-21T20:01:41.923 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:01:42.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:42 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:01:41.148306+0000 mgr.smithi053.fgsmol (mgr.14164) 77 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:42.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:42 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:01:41.930683+0000 mon.smithi053 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.146:0/2305026890' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:01:43.311 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:01:43.312 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:01:44.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:44.556+0000 7f465e49d700 1 -- 172.21.15.146:0/2787592595 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46580f5410 msgr2=0x7f46580f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:44.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:44.556+0000 7f465e49d700 1 --2- 172.21.15.146:0/2787592595 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46580f5410 0x7f46580f5830 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f4648009fb0 tx=0x7f4648009b30).stop 2024-01-21T20:01:44.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:44.556+0000 7f465e49d700 1 -- 172.21.15.146:0/2787592595 shutdown_connections 2024-01-21T20:01:44.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:44.556+0000 7f465e49d700 1 --2- 172.21.15.146:0/2787592595 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46580f5410 0x7f46580f5830 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:44.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:44.556+0000 7f465e49d700 1 -- 172.21.15.146:0/2787592595 >> 172.21.15.146:0/2787592595 conn(0x7f46580f0b70 msgr2=0x7f46580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:44.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:44.556+0000 7f465e49d700 1 -- 172.21.15.146:0/2787592595 shutdown_connections 2024-01-21T20:01:44.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:44.556+0000 7f465e49d700 1 -- 172.21.15.146:0/2787592595 wait complete. 2024-01-21T20:01:44.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:44.556+0000 7f465e49d700 1 Processor -- start 2024-01-21T20:01:44.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:44.557+0000 7f465e49d700 1 -- start start 2024-01-21T20:01:44.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:44.557+0000 7f465e49d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46580f5410 0x7f4658106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:44.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:44.557+0000 7f465e49d700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4658106c20 con 0x7f46580f5410 2024-01-21T20:01:44.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:44.557+0000 7f4657fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46580f5410 0x7f4658106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:44.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:44.557+0000 7f4657fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46580f5410 0x7f4658106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:33048/0 (socket says 172.21.15.146:33048) 2024-01-21T20:01:44.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:44.557+0000 7f4657fff700 1 -- 172.21.15.146:0/3493199559 learned_addr learned my addr 172.21.15.146:0/3493199559 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:01:44.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:44.558+0000 7f4657fff700 1 -- 172.21.15.146:0/3493199559 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4648005040 con 0x7f46580f5410 2024-01-21T20:01:44.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:44.558+0000 7f4657fff700 1 --2- 172.21.15.146:0/3493199559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46580f5410 0x7f4658106730 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f4648013510 tx=0x7f464800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:44.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:44.558+0000 7f4654ff9700 1 -- 172.21.15.146:0/3493199559 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4648009320 con 0x7f46580f5410 2024-01-21T20:01:44.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:44.558+0000 7f465e49d700 1 -- 172.21.15.146:0/3493199559 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4658106e20 con 0x7f46580f5410 2024-01-21T20:01:44.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:44.558+0000 7f4654ff9700 1 -- 172.21.15.146:0/3493199559 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4648013f00 con 0x7f46580f5410 2024-01-21T20:01:44.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:44.558+0000 7f4654ff9700 1 -- 172.21.15.146:0/3493199559 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4648014d90 con 0x7f46580f5410 2024-01-21T20:01:44.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:44.558+0000 7f465e49d700 1 -- 172.21.15.146:0/3493199559 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46581072b0 con 0x7f46580f5410 2024-01-21T20:01:44.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:44.559+0000 7f4654ff9700 1 -- 172.21.15.146:0/3493199559 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f46480095f0 con 0x7f46580f5410 2024-01-21T20:01:44.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:44.559+0000 7f465e49d700 1 -- 172.21.15.146:0/3493199559 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4658045ad0 con 0x7f46580f5410 2024-01-21T20:01:44.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:44.559+0000 7f4654ff9700 1 --2- 172.21.15.146:0/3493199559 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f464002ff60 0x7f4640032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:44.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:44.559+0000 7f4654ff9700 1 -- 172.21.15.146:0/3493199559 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4648041000 con 0x7f46580f5410 2024-01-21T20:01:44.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:44.559+0000 7f46577fe700 1 --2- 172.21.15.146:0/3493199559 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f464002ff60 0x7f4640032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:44.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:44.560+0000 7f46577fe700 1 --2- 172.21.15.146:0/3493199559 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f464002ff60 0x7f4640032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f464c0096a0 tx=0x7f464c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:44.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:44.564+0000 7f4654ff9700 1 -- 172.21.15.146:0/3493199559 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4648014420 con 0x7f46580f5410 2024-01-21T20:01:44.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:44.767+0000 7f465e49d700 1 -- 172.21.15.146:0/3493199559 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f46580fae60 con 0x7f46580f5410 2024-01-21T20:01:44.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:44.768+0000 7f4654ff9700 1 -- 172.21.15.146:0/3493199559 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4648010230 con 0x7f46580f5410 2024-01-21T20:01:44.760 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:01:44.760 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:01:44.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:44.770+0000 7f465e49d700 1 -- 172.21.15.146:0/3493199559 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f464002ff60 msgr2=0x7f4640032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:44.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:44.770+0000 7f465e49d700 1 --2- 172.21.15.146:0/3493199559 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f464002ff60 0x7f4640032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f464c0096a0 tx=0x7f464c006b40).stop 2024-01-21T20:01:44.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:44.770+0000 7f465e49d700 1 -- 172.21.15.146:0/3493199559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46580f5410 msgr2=0x7f4658106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:44.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:44.770+0000 7f465e49d700 1 --2- 172.21.15.146:0/3493199559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46580f5410 0x7f4658106730 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f4648013510 tx=0x7f464800b040).stop 2024-01-21T20:01:44.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:44.770+0000 7f465e49d700 1 -- 172.21.15.146:0/3493199559 shutdown_connections 2024-01-21T20:01:44.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:44.770+0000 7f465e49d700 1 --2- 172.21.15.146:0/3493199559 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f464002ff60 0x7f4640032420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:44.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:44.770+0000 7f465e49d700 1 --2- 172.21.15.146:0/3493199559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46580f5410 0x7f4658106730 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:44.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:44.770+0000 7f465e49d700 1 -- 172.21.15.146:0/3493199559 >> 172.21.15.146:0/3493199559 conn(0x7f46580f0b70 msgr2=0x7f46580f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:44.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:44.770+0000 7f465e49d700 1 -- 172.21.15.146:0/3493199559 shutdown_connections 2024-01-21T20:01:44.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:44.770+0000 7f465e49d700 1 -- 172.21.15.146:0/3493199559 wait complete. 2024-01-21T20:01:44.762 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:01:44.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:44 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:01:43.148632+0000 mgr.smithi053.fgsmol (mgr.14164) 78 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:45.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:45 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:01:44.769501+0000 mon.smithi053 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.146:0/3493199559' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:01:46.167 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:01:46.167 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:01:46.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:46 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:01:45.148989+0000 mgr.smithi053.fgsmol (mgr.14164) 79 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:47.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:47.441+0000 7f321f2f0700 1 -- 172.21.15.146:0/3401687192 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32180f6510 msgr2=0x7f32180f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:47.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:47.441+0000 7f321f2f0700 1 --2- 172.21.15.146:0/3401687192 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32180f6510 0x7f32180f6930 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f3208009fb0 tx=0x7f3208009b30).stop 2024-01-21T20:01:47.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:47.441+0000 7f321f2f0700 1 -- 172.21.15.146:0/3401687192 shutdown_connections 2024-01-21T20:01:47.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:47.441+0000 7f321f2f0700 1 --2- 172.21.15.146:0/3401687192 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32180f6510 0x7f32180f6930 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:47.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:47.441+0000 7f321f2f0700 1 -- 172.21.15.146:0/3401687192 >> 172.21.15.146:0/3401687192 conn(0x7f32180f1c70 msgr2=0x7f32180f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:47.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:47.441+0000 7f321f2f0700 1 -- 172.21.15.146:0/3401687192 shutdown_connections 2024-01-21T20:01:47.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:47.441+0000 7f321f2f0700 1 -- 172.21.15.146:0/3401687192 wait complete. 2024-01-21T20:01:47.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:47.442+0000 7f321f2f0700 1 Processor -- start 2024-01-21T20:01:47.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:47.442+0000 7f321f2f0700 1 -- start start 2024-01-21T20:01:47.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:47.442+0000 7f321f2f0700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32180f6510 0x7f3218106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:47.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:47.442+0000 7f321f2f0700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3218106c20 con 0x7f32180f6510 2024-01-21T20:01:47.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:47.443+0000 7f321d08c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32180f6510 0x7f3218106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:47.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:47.443+0000 7f321d08c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32180f6510 0x7f3218106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:60610/0 (socket says 172.21.15.146:60610) 2024-01-21T20:01:47.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:47.443+0000 7f321d08c700 1 -- 172.21.15.146:0/299366409 learned_addr learned my addr 172.21.15.146:0/299366409 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:01:47.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:47.443+0000 7f321d08c700 1 -- 172.21.15.146:0/299366409 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3208005040 con 0x7f32180f6510 2024-01-21T20:01:47.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:47.443+0000 7f321d08c700 1 --2- 172.21.15.146:0/299366409 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32180f6510 0x7f3218106730 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f3208013510 tx=0x7f320800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:47.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:47.443+0000 7f320dffb700 1 -- 172.21.15.146:0/299366409 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3208009320 con 0x7f32180f6510 2024-01-21T20:01:47.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:47.443+0000 7f321f2f0700 1 -- 172.21.15.146:0/299366409 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3218106e20 con 0x7f32180f6510 2024-01-21T20:01:47.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:47.443+0000 7f320dffb700 1 -- 172.21.15.146:0/299366409 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3208013f00 con 0x7f32180f6510 2024-01-21T20:01:47.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:47.443+0000 7f320dffb700 1 -- 172.21.15.146:0/299366409 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3208014d90 con 0x7f32180f6510 2024-01-21T20:01:47.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:47.443+0000 7f321f2f0700 1 -- 172.21.15.146:0/299366409 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32181072b0 con 0x7f32180f6510 2024-01-21T20:01:47.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:47.444+0000 7f320dffb700 1 -- 172.21.15.146:0/299366409 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f32080095f0 con 0x7f32180f6510 2024-01-21T20:01:47.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:47.444+0000 7f321f2f0700 1 -- 172.21.15.146:0/299366409 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3218045ad0 con 0x7f32180f6510 2024-01-21T20:01:47.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:47.444+0000 7f320dffb700 1 --2- 172.21.15.146:0/299366409 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f320402ff60 0x7f3204032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:47.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:47.444+0000 7f320dffb700 1 -- 172.21.15.146:0/299366409 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3208041080 con 0x7f32180f6510 2024-01-21T20:01:47.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:47.445+0000 7f321c88b700 1 --2- 172.21.15.146:0/299366409 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f320402ff60 0x7f3204032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:47.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:47.445+0000 7f321c88b700 1 --2- 172.21.15.146:0/299366409 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f320402ff60 0x7f3204032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f32140096a0 tx=0x7f3214006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:47.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:47.450+0000 7f320dffb700 1 -- 172.21.15.146:0/299366409 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3208014420 con 0x7f32180f6510 2024-01-21T20:01:47.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:47.652+0000 7f321f2f0700 1 -- 172.21.15.146:0/299366409 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f32180fbf60 con 0x7f32180f6510 2024-01-21T20:01:47.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:47.653+0000 7f320dffb700 1 -- 172.21.15.146:0/299366409 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3208010230 con 0x7f32180f6510 2024-01-21T20:01:47.644 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:01:47.644 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:01:47.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:47.654+0000 7f321f2f0700 1 -- 172.21.15.146:0/299366409 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f320402ff60 msgr2=0x7f3204032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:47.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:47.654+0000 7f321f2f0700 1 --2- 172.21.15.146:0/299366409 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f320402ff60 0x7f3204032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f32140096a0 tx=0x7f3214006b40).stop 2024-01-21T20:01:47.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:47.654+0000 7f321f2f0700 1 -- 172.21.15.146:0/299366409 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32180f6510 msgr2=0x7f3218106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:47.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:47.654+0000 7f321f2f0700 1 --2- 172.21.15.146:0/299366409 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32180f6510 0x7f3218106730 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f3208013510 tx=0x7f320800b040).stop 2024-01-21T20:01:47.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:47.654+0000 7f321f2f0700 1 -- 172.21.15.146:0/299366409 shutdown_connections 2024-01-21T20:01:47.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:47.654+0000 7f321f2f0700 1 --2- 172.21.15.146:0/299366409 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f320402ff60 0x7f3204032420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:47.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:47.654+0000 7f321f2f0700 1 --2- 172.21.15.146:0/299366409 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32180f6510 0x7f3218106730 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:47.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:47.654+0000 7f321f2f0700 1 -- 172.21.15.146:0/299366409 >> 172.21.15.146:0/299366409 conn(0x7f32180f1c70 msgr2=0x7f32180f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:47.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:47.654+0000 7f321f2f0700 1 -- 172.21.15.146:0/299366409 shutdown_connections 2024-01-21T20:01:47.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:47.654+0000 7f321f2f0700 1 -- 172.21.15.146:0/299366409 wait complete. 2024-01-21T20:01:47.647 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:01:48.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:48 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:01:47.149336+0000 mgr.smithi053.fgsmol (mgr.14164) 80 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:48.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:48 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:01:47.654077+0000 mon.smithi053 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.146:0/299366409' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:01:49.080 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:01:49.081 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:01:50.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:50.331+0000 7fb63895e700 1 -- 172.21.15.146:0/2330376627 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6340f51a0 msgr2=0x7fb6340f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:50.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:50.331+0000 7fb63895e700 1 --2- 172.21.15.146:0/2330376627 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6340f51a0 0x7fb6340f55c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fb624009fb0 tx=0x7fb624009b30).stop 2024-01-21T20:01:50.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:50.331+0000 7fb63895e700 1 -- 172.21.15.146:0/2330376627 shutdown_connections 2024-01-21T20:01:50.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:50.331+0000 7fb63895e700 1 --2- 172.21.15.146:0/2330376627 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6340f51a0 0x7fb6340f55c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:50.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:50.331+0000 7fb63895e700 1 -- 172.21.15.146:0/2330376627 >> 172.21.15.146:0/2330376627 conn(0x7fb6340f09a0 msgr2=0x7fb6340f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:50.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:50.331+0000 7fb63895e700 1 -- 172.21.15.146:0/2330376627 shutdown_connections 2024-01-21T20:01:50.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:50.331+0000 7fb63895e700 1 -- 172.21.15.146:0/2330376627 wait complete. 2024-01-21T20:01:50.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:50.331+0000 7fb63895e700 1 Processor -- start 2024-01-21T20:01:50.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:50.331+0000 7fb63895e700 1 -- start start 2024-01-21T20:01:50.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:50.332+0000 7fb63895e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6340f51a0 0x7fb6341053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:50.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:50.332+0000 7fb63895e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6341058a0 con 0x7fb6340f51a0 2024-01-21T20:01:50.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:50.332+0000 7fb632d9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6340f51a0 0x7fb6341053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:50.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:50.332+0000 7fb632d9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6340f51a0 0x7fb6341053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:60624/0 (socket says 172.21.15.146:60624) 2024-01-21T20:01:50.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:50.332+0000 7fb632d9d700 1 -- 172.21.15.146:0/1746535150 learned_addr learned my addr 172.21.15.146:0/1746535150 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:01:50.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:50.332+0000 7fb632d9d700 1 -- 172.21.15.146:0/1746535150 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb624005040 con 0x7fb6340f51a0 2024-01-21T20:01:50.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:50.332+0000 7fb632d9d700 1 --2- 172.21.15.146:0/1746535150 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6340f51a0 0x7fb6341053b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fb624013510 tx=0x7fb62400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:50.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:50.332+0000 7fb61b7fe700 1 -- 172.21.15.146:0/1746535150 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb624009320 con 0x7fb6340f51a0 2024-01-21T20:01:50.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:50.333+0000 7fb63895e700 1 -- 172.21.15.146:0/1746535150 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb634105aa0 con 0x7fb6340f51a0 2024-01-21T20:01:50.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:50.333+0000 7fb61b7fe700 1 -- 172.21.15.146:0/1746535150 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb624013f00 con 0x7fb6340f51a0 2024-01-21T20:01:50.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:50.333+0000 7fb61b7fe700 1 -- 172.21.15.146:0/1746535150 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb624014dc0 con 0x7fb6340f51a0 2024-01-21T20:01:50.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:50.333+0000 7fb63895e700 1 -- 172.21.15.146:0/1746535150 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb634105f30 con 0x7fb6340f51a0 2024-01-21T20:01:50.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:50.333+0000 7fb61b7fe700 1 -- 172.21.15.146:0/1746535150 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb6240095f0 con 0x7fb6340f51a0 2024-01-21T20:01:50.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:50.333+0000 7fb63895e700 1 -- 172.21.15.146:0/1746535150 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb634045ad0 con 0x7fb6340f51a0 2024-01-21T20:01:50.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:50.333+0000 7fb61b7fe700 1 --2- 172.21.15.146:0/1746535150 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb61c02ff60 0x7fb61c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:50.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:50.334+0000 7fb61b7fe700 1 -- 172.21.15.146:0/1746535150 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb624041000 con 0x7fb6340f51a0 2024-01-21T20:01:50.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:50.334+0000 7fb63259c700 1 --2- 172.21.15.146:0/1746535150 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb61c02ff60 0x7fb61c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:50.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:50.334+0000 7fb63259c700 1 --2- 172.21.15.146:0/1746535150 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb61c02ff60 0x7fb61c032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fb6280096a0 tx=0x7fb628006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:50.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:50.338+0000 7fb61b7fe700 1 -- 172.21.15.146:0/1746535150 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb624014420 con 0x7fb6340f51a0 2024-01-21T20:01:50.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:50.544+0000 7fb63895e700 1 -- 172.21.15.146:0/1746535150 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb634106de0 con 0x7fb6340f51a0 2024-01-21T20:01:50.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:50.544+0000 7fb61b7fe700 1 -- 172.21.15.146:0/1746535150 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb624010230 con 0x7fb6340f51a0 2024-01-21T20:01:50.536 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:01:50.536 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:01:50.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:50.546+0000 7fb63895e700 1 -- 172.21.15.146:0/1746535150 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb61c02ff60 msgr2=0x7fb61c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:50.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:50.546+0000 7fb63895e700 1 --2- 172.21.15.146:0/1746535150 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb61c02ff60 0x7fb61c032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fb6280096a0 tx=0x7fb628006b40).stop 2024-01-21T20:01:50.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:50.546+0000 7fb63895e700 1 -- 172.21.15.146:0/1746535150 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6340f51a0 msgr2=0x7fb6341053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:50.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:50.546+0000 7fb63895e700 1 --2- 172.21.15.146:0/1746535150 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6340f51a0 0x7fb6341053b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fb624013510 tx=0x7fb62400b040).stop 2024-01-21T20:01:50.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:50.546+0000 7fb63895e700 1 -- 172.21.15.146:0/1746535150 shutdown_connections 2024-01-21T20:01:50.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:50.546+0000 7fb63895e700 1 --2- 172.21.15.146:0/1746535150 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb61c02ff60 0x7fb61c032420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:50.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:50.546+0000 7fb63895e700 1 --2- 172.21.15.146:0/1746535150 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6340f51a0 0x7fb6341053b0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:50.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:50.546+0000 7fb63895e700 1 -- 172.21.15.146:0/1746535150 >> 172.21.15.146:0/1746535150 conn(0x7fb6340f09a0 msgr2=0x7fb6340f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:50.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:50.546+0000 7fb63895e700 1 -- 172.21.15.146:0/1746535150 shutdown_connections 2024-01-21T20:01:50.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:50.546+0000 7fb63895e700 1 -- 172.21.15.146:0/1746535150 wait complete. 2024-01-21T20:01:50.538 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:01:50.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:50 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:01:49.149735+0000 mgr.smithi053.fgsmol (mgr.14164) 81 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:50.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:50 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:01:50.545830+0000 mon.smithi053 (mon.0) 167 : audit [DBG] from='client.? 172.21.15.146:0/1746535150' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:01:51.962 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:01:51.963 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:01:52.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:52 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:01:51.150024+0000 mgr.smithi053.fgsmol (mgr.14164) 82 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:53.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:53.251+0000 7fe10e6f3700 1 -- 172.21.15.146:0/586640264 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1080f62a0 msgr2=0x7fe1080f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:53.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:53.251+0000 7fe10e6f3700 1 --2- 172.21.15.146:0/586640264 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1080f62a0 0x7fe1080f66c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fe0f8009fb0 tx=0x7fe0f8009b30).stop 2024-01-21T20:01:53.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:53.251+0000 7fe10e6f3700 1 -- 172.21.15.146:0/586640264 shutdown_connections 2024-01-21T20:01:53.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:53.251+0000 7fe10e6f3700 1 --2- 172.21.15.146:0/586640264 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1080f62a0 0x7fe1080f66c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:53.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:53.251+0000 7fe10e6f3700 1 -- 172.21.15.146:0/586640264 >> 172.21.15.146:0/586640264 conn(0x7fe1080f1aa0 msgr2=0x7fe1080f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:53.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:53.251+0000 7fe10e6f3700 1 -- 172.21.15.146:0/586640264 shutdown_connections 2024-01-21T20:01:53.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:53.251+0000 7fe10e6f3700 1 -- 172.21.15.146:0/586640264 wait complete. 2024-01-21T20:01:53.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:53.251+0000 7fe10e6f3700 1 Processor -- start 2024-01-21T20:01:53.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:53.251+0000 7fe10e6f3700 1 -- start start 2024-01-21T20:01:53.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:53.252+0000 7fe10e6f3700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1080f62a0 0x7fe1081064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:53.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:53.252+0000 7fe10e6f3700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1081069a0 con 0x7fe1080f62a0 2024-01-21T20:01:53.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:53.252+0000 7fe10d6f1700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1080f62a0 0x7fe1081064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:53.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:53.252+0000 7fe10d6f1700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1080f62a0 0x7fe1081064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:60638/0 (socket says 172.21.15.146:60638) 2024-01-21T20:01:53.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:53.252+0000 7fe10d6f1700 1 -- 172.21.15.146:0/745787371 learned_addr learned my addr 172.21.15.146:0/745787371 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:01:53.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:53.252+0000 7fe10d6f1700 1 -- 172.21.15.146:0/745787371 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f8005040 con 0x7fe1080f62a0 2024-01-21T20:01:53.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:53.252+0000 7fe10d6f1700 1 --2- 172.21.15.146:0/745787371 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1080f62a0 0x7fe1081064b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fe0f8013510 tx=0x7fe0f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:53.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:53.252+0000 7fe0fdffb700 1 -- 172.21.15.146:0/745787371 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0f8009320 con 0x7fe1080f62a0 2024-01-21T20:01:53.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:53.252+0000 7fe10e6f3700 1 -- 172.21.15.146:0/745787371 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe108106ba0 con 0x7fe1080f62a0 2024-01-21T20:01:53.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:53.252+0000 7fe0fdffb700 1 -- 172.21.15.146:0/745787371 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe0f8013f00 con 0x7fe1080f62a0 2024-01-21T20:01:53.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:53.252+0000 7fe0fdffb700 1 -- 172.21.15.146:0/745787371 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0f8014dc0 con 0x7fe1080f62a0 2024-01-21T20:01:53.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:53.253+0000 7fe10e6f3700 1 -- 172.21.15.146:0/745787371 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe108107030 con 0x7fe1080f62a0 2024-01-21T20:01:53.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:53.253+0000 7fe0fdffb700 1 -- 172.21.15.146:0/745787371 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe0f8014420 con 0x7fe1080f62a0 2024-01-21T20:01:53.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:53.253+0000 7fe10e6f3700 1 -- 172.21.15.146:0/745787371 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe108045ad0 con 0x7fe1080f62a0 2024-01-21T20:01:53.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:53.254+0000 7fe0fdffb700 1 --2- 172.21.15.146:0/745787371 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fe0f402ff60 0x7fe0f4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:53.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:53.254+0000 7fe0fdffb700 1 -- 172.21.15.146:0/745787371 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe0f80410b0 con 0x7fe1080f62a0 2024-01-21T20:01:53.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:53.254+0000 7fe10cef0700 1 --2- 172.21.15.146:0/745787371 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fe0f402ff60 0x7fe0f4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:53.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:53.254+0000 7fe10cef0700 1 --2- 172.21.15.146:0/745787371 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fe0f402ff60 0x7fe0f4032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fe1040096a0 tx=0x7fe104006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:53.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:53.258+0000 7fe0fdffb700 1 -- 172.21.15.146:0/745787371 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe0f80095f0 con 0x7fe1080f62a0 2024-01-21T20:01:53.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:53.463+0000 7fe10e6f3700 1 -- 172.21.15.146:0/745787371 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe108107ee0 con 0x7fe1080f62a0 2024-01-21T20:01:53.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:53.464+0000 7fe0fdffb700 1 -- 172.21.15.146:0/745787371 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe0f801cb10 con 0x7fe1080f62a0 2024-01-21T20:01:53.455 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:01:53.456 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:01:53.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:53.465+0000 7fe10e6f3700 1 -- 172.21.15.146:0/745787371 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fe0f402ff60 msgr2=0x7fe0f4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:53.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:53.466+0000 7fe10e6f3700 1 --2- 172.21.15.146:0/745787371 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fe0f402ff60 0x7fe0f4032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fe1040096a0 tx=0x7fe104006b40).stop 2024-01-21T20:01:53.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:53.466+0000 7fe10e6f3700 1 -- 172.21.15.146:0/745787371 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1080f62a0 msgr2=0x7fe1081064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:53.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:53.466+0000 7fe10e6f3700 1 --2- 172.21.15.146:0/745787371 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1080f62a0 0x7fe1081064b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fe0f8013510 tx=0x7fe0f800b040).stop 2024-01-21T20:01:53.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:53.466+0000 7fe10e6f3700 1 -- 172.21.15.146:0/745787371 shutdown_connections 2024-01-21T20:01:53.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:53.466+0000 7fe10e6f3700 1 --2- 172.21.15.146:0/745787371 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fe0f402ff60 0x7fe0f4032420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:53.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:53.466+0000 7fe10e6f3700 1 --2- 172.21.15.146:0/745787371 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1080f62a0 0x7fe1081064b0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:53.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:53.466+0000 7fe10e6f3700 1 -- 172.21.15.146:0/745787371 >> 172.21.15.146:0/745787371 conn(0x7fe1080f1aa0 msgr2=0x7fe1080f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:53.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:53.466+0000 7fe10e6f3700 1 -- 172.21.15.146:0/745787371 shutdown_connections 2024-01-21T20:01:53.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:53.466+0000 7fe10e6f3700 1 -- 172.21.15.146:0/745787371 wait complete. 2024-01-21T20:01:53.458 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:01:53.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:53 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:01:53.465018+0000 mon.smithi053 (mon.0) 168 : audit [DBG] from='client.? 172.21.15.146:0/745787371' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:01:54.868 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:01:54.868 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:01:54.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:54 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:01:53.150319+0000 mgr.smithi053.fgsmol (mgr.14164) 83 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:56.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:56.073+0000 7f346a1c6700 1 -- 172.21.15.146:0/1458577203 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34640f5410 msgr2=0x7f34640f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:56.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:56.073+0000 7f346a1c6700 1 --2- 172.21.15.146:0/1458577203 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34640f5410 0x7f34640f5830 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f3454009fb0 tx=0x7f3454009b30).stop 2024-01-21T20:01:56.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:56.073+0000 7f346a1c6700 1 -- 172.21.15.146:0/1458577203 shutdown_connections 2024-01-21T20:01:56.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:56.073+0000 7f346a1c6700 1 --2- 172.21.15.146:0/1458577203 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34640f5410 0x7f34640f5830 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:56.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:56.073+0000 7f346a1c6700 1 -- 172.21.15.146:0/1458577203 >> 172.21.15.146:0/1458577203 conn(0x7f34640f0b70 msgr2=0x7f34640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:56.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:56.074+0000 7f346a1c6700 1 -- 172.21.15.146:0/1458577203 shutdown_connections 2024-01-21T20:01:56.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:56.074+0000 7f346a1c6700 1 -- 172.21.15.146:0/1458577203 wait complete. 2024-01-21T20:01:56.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:56.074+0000 7f346a1c6700 1 Processor -- start 2024-01-21T20:01:56.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:56.074+0000 7f346a1c6700 1 -- start start 2024-01-21T20:01:56.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:56.074+0000 7f346a1c6700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34640f5410 0x7f3464105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:56.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:56.074+0000 7f346a1c6700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3464105af0 con 0x7f34640f5410 2024-01-21T20:01:56.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:56.075+0000 7f34637fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34640f5410 0x7f3464105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:56.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:56.075+0000 7f34637fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34640f5410 0x7f3464105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:60648/0 (socket says 172.21.15.146:60648) 2024-01-21T20:01:56.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:56.075+0000 7f34637fe700 1 -- 172.21.15.146:0/2330160539 learned_addr learned my addr 172.21.15.146:0/2330160539 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:01:56.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:56.075+0000 7f34637fe700 1 -- 172.21.15.146:0/2330160539 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3454005040 con 0x7f34640f5410 2024-01-21T20:01:56.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:56.075+0000 7f34637fe700 1 --2- 172.21.15.146:0/2330160539 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34640f5410 0x7f3464105600 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f3454013510 tx=0x7f345400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:56.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:56.075+0000 7f344bfff700 1 -- 172.21.15.146:0/2330160539 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3454009320 con 0x7f34640f5410 2024-01-21T20:01:56.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:56.075+0000 7f346a1c6700 1 -- 172.21.15.146:0/2330160539 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3464105cf0 con 0x7f34640f5410 2024-01-21T20:01:56.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:56.075+0000 7f344bfff700 1 -- 172.21.15.146:0/2330160539 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3454013f00 con 0x7f34640f5410 2024-01-21T20:01:56.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:56.075+0000 7f344bfff700 1 -- 172.21.15.146:0/2330160539 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3454014d90 con 0x7f34640f5410 2024-01-21T20:01:56.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:56.075+0000 7f346a1c6700 1 -- 172.21.15.146:0/2330160539 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3464106180 con 0x7f34640f5410 2024-01-21T20:01:56.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:56.076+0000 7f344bfff700 1 -- 172.21.15.146:0/2330160539 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f34540263a0 con 0x7f34640f5410 2024-01-21T20:01:56.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:56.076+0000 7f344bfff700 1 --2- 172.21.15.146:0/2330160539 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f344c02ffb0 0x7f344c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:56.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:56.076+0000 7f344bfff700 1 -- 172.21.15.146:0/2330160539 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3454041110 con 0x7f34640f5410 2024-01-21T20:01:56.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:56.076+0000 7f346a1c6700 1 -- 172.21.15.146:0/2330160539 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3464045ad0 con 0x7f34640f5410 2024-01-21T20:01:56.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:56.077+0000 7f3462ffd700 1 --2- 172.21.15.146:0/2330160539 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f344c02ffb0 0x7f344c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:56.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:56.077+0000 7f3462ffd700 1 --2- 172.21.15.146:0/2330160539 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f344c02ffb0 0x7f344c032470 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f34580096a0 tx=0x7f3458006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:56.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:56.081+0000 7f344bfff700 1 -- 172.21.15.146:0/2330160539 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3454012920 con 0x7f34640f5410 2024-01-21T20:01:56.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:56.286+0000 7f346a1c6700 1 -- 172.21.15.146:0/2330160539 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f34640faf50 con 0x7f34640f5410 2024-01-21T20:01:56.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:56.287+0000 7f344bfff700 1 -- 172.21.15.146:0/2330160539 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f345402b020 con 0x7f34640f5410 2024-01-21T20:01:56.278 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:01:56.278 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:01:56.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:56.288+0000 7f346a1c6700 1 -- 172.21.15.146:0/2330160539 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f344c02ffb0 msgr2=0x7f344c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:56.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:56.288+0000 7f346a1c6700 1 --2- 172.21.15.146:0/2330160539 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f344c02ffb0 0x7f344c032470 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f34580096a0 tx=0x7f3458006b40).stop 2024-01-21T20:01:56.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:56.289+0000 7f346a1c6700 1 -- 172.21.15.146:0/2330160539 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34640f5410 msgr2=0x7f3464105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:56.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:56.289+0000 7f346a1c6700 1 --2- 172.21.15.146:0/2330160539 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34640f5410 0x7f3464105600 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f3454013510 tx=0x7f345400b040).stop 2024-01-21T20:01:56.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:56.289+0000 7f346a1c6700 1 -- 172.21.15.146:0/2330160539 shutdown_connections 2024-01-21T20:01:56.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:56.289+0000 7f346a1c6700 1 --2- 172.21.15.146:0/2330160539 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f344c02ffb0 0x7f344c032470 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:56.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:56.289+0000 7f346a1c6700 1 --2- 172.21.15.146:0/2330160539 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34640f5410 0x7f3464105600 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:56.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:56.289+0000 7f346a1c6700 1 -- 172.21.15.146:0/2330160539 >> 172.21.15.146:0/2330160539 conn(0x7f34640f0b70 msgr2=0x7f34640f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:56.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:56.289+0000 7f346a1c6700 1 -- 172.21.15.146:0/2330160539 shutdown_connections 2024-01-21T20:01:56.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:56.289+0000 7f346a1c6700 1 -- 172.21.15.146:0/2330160539 wait complete. 2024-01-21T20:01:56.281 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:01:56.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:56 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:01:55.150599+0000 mgr.smithi053.fgsmol (mgr.14164) 84 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:56.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:56 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:01:56.288200+0000 mon.smithi053 (mon.0) 169 : audit [DBG] from='client.? 172.21.15.146:0/2330160539' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:01:57.679 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:01:57.679 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:01:58.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:58.921+0000 7f79e54f4700 1 -- 172.21.15.146:0/3233794550 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79e00f5410 msgr2=0x7f79e00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:58.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:58.921+0000 7f79e54f4700 1 --2- 172.21.15.146:0/3233794550 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79e00f5410 0x7f79e00f5830 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f79d0009fb0 tx=0x7f79d0009b30).stop 2024-01-21T20:01:58.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:58.921+0000 7f79e54f4700 1 -- 172.21.15.146:0/3233794550 shutdown_connections 2024-01-21T20:01:58.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:58.921+0000 7f79e54f4700 1 --2- 172.21.15.146:0/3233794550 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79e00f5410 0x7f79e00f5830 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:58.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:58.921+0000 7f79e54f4700 1 -- 172.21.15.146:0/3233794550 >> 172.21.15.146:0/3233794550 conn(0x7f79e00f0b70 msgr2=0x7f79e00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:58.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:58.921+0000 7f79e54f4700 1 -- 172.21.15.146:0/3233794550 shutdown_connections 2024-01-21T20:01:58.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:58.921+0000 7f79e54f4700 1 -- 172.21.15.146:0/3233794550 wait complete. 2024-01-21T20:01:58.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:58.921+0000 7f79e54f4700 1 Processor -- start 2024-01-21T20:01:58.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:58.922+0000 7f79e54f4700 1 -- start start 2024-01-21T20:01:58.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:58.922+0000 7f79e54f4700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79e00f5410 0x7f79e0109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:58.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:58.922+0000 7f79e54f4700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e0060240 con 0x7f79e00f5410 2024-01-21T20:01:58.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:58.922+0000 7f79deffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79e00f5410 0x7f79e0109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:58.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:58.922+0000 7f79deffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79e00f5410 0x7f79e0109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:56960/0 (socket says 172.21.15.146:56960) 2024-01-21T20:01:58.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:58.922+0000 7f79deffd700 1 -- 172.21.15.146:0/1140770502 learned_addr learned my addr 172.21.15.146:0/1140770502 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:01:58.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:58.922+0000 7f79deffd700 1 -- 172.21.15.146:0/1140770502 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d0005040 con 0x7f79e00f5410 2024-01-21T20:01:58.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:58.922+0000 7f79deffd700 1 --2- 172.21.15.146:0/1140770502 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79e00f5410 0x7f79e0109400 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f79d0014000 tx=0x7f79d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:58.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:58.923+0000 7f79c77fe700 1 -- 172.21.15.146:0/1140770502 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f79d00095f0 con 0x7f79e00f5410 2024-01-21T20:01:58.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:58.923+0000 7f79e54f4700 1 -- 172.21.15.146:0/1140770502 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79e0060440 con 0x7f79e00f5410 2024-01-21T20:01:58.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:58.923+0000 7f79c77fe700 1 -- 172.21.15.146:0/1140770502 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f79d0007e70 con 0x7f79e00f5410 2024-01-21T20:01:58.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:58.923+0000 7f79c77fe700 1 -- 172.21.15.146:0/1140770502 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f79d0014ea0 con 0x7f79e00f5410 2024-01-21T20:01:58.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:58.923+0000 7f79e54f4700 1 -- 172.21.15.146:0/1140770502 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79e0109ab0 con 0x7f79e00f5410 2024-01-21T20:01:58.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:58.923+0000 7f79c77fe700 1 -- 172.21.15.146:0/1140770502 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f79d00079c0 con 0x7f79e00f5410 2024-01-21T20:01:58.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:58.923+0000 7f79e54f4700 1 -- 172.21.15.146:0/1140770502 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79e0045ad0 con 0x7f79e00f5410 2024-01-21T20:01:58.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:58.924+0000 7f79c77fe700 1 --2- 172.21.15.146:0/1140770502 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f79c802ff60 0x7f79c8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:58.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:58.924+0000 7f79c77fe700 1 -- 172.21.15.146:0/1140770502 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f79d0040860 con 0x7f79e00f5410 2024-01-21T20:01:58.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:58.924+0000 7f79de7fc700 1 --2- 172.21.15.146:0/1140770502 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f79c802ff60 0x7f79c8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:58.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:58.924+0000 7f79de7fc700 1 --2- 172.21.15.146:0/1140770502 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f79c802ff60 0x7f79c8032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f79d40096a0 tx=0x7f79d4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:58.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:58.929+0000 7f79c77fe700 1 -- 172.21.15.146:0/1140770502 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f79d0010230 con 0x7f79e00f5410 2024-01-21T20:01:58.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:58 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:01:57.150932+0000 mgr.smithi053.fgsmol (mgr.14164) 85 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:59.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:59.137+0000 7f79e54f4700 1 -- 172.21.15.146:0/1140770502 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f79e00fae60 con 0x7f79e00f5410 2024-01-21T20:01:59.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:59.137+0000 7f79c77fe700 1 -- 172.21.15.146:0/1140770502 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f79d0010230 con 0x7f79e00f5410 2024-01-21T20:01:59.129 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:01:59.129 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:01:59.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:59.139+0000 7f79e54f4700 1 -- 172.21.15.146:0/1140770502 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f79c802ff60 msgr2=0x7f79c8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:59.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:59.139+0000 7f79e54f4700 1 --2- 172.21.15.146:0/1140770502 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f79c802ff60 0x7f79c8032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f79d40096a0 tx=0x7f79d4006b40).stop 2024-01-21T20:01:59.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:59.139+0000 7f79e54f4700 1 -- 172.21.15.146:0/1140770502 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79e00f5410 msgr2=0x7f79e0109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:59.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:59.139+0000 7f79e54f4700 1 --2- 172.21.15.146:0/1140770502 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79e00f5410 0x7f79e0109400 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f79d0014000 tx=0x7f79d000b040).stop 2024-01-21T20:01:59.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:59.139+0000 7f79e54f4700 1 -- 172.21.15.146:0/1140770502 shutdown_connections 2024-01-21T20:01:59.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:59.139+0000 7f79e54f4700 1 --2- 172.21.15.146:0/1140770502 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f79c802ff60 0x7f79c8032420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:59.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:59.139+0000 7f79e54f4700 1 --2- 172.21.15.146:0/1140770502 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79e00f5410 0x7f79e0109400 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:59.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:59.139+0000 7f79e54f4700 1 -- 172.21.15.146:0/1140770502 >> 172.21.15.146:0/1140770502 conn(0x7f79e00f0b70 msgr2=0x7f79e00f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:59.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:59.139+0000 7f79e54f4700 1 -- 172.21.15.146:0/1140770502 shutdown_connections 2024-01-21T20:01:59.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:01:59.139+0000 7f79e54f4700 1 -- 172.21.15.146:0/1140770502 wait complete. 2024-01-21T20:01:59.132 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:01:59.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:01:59 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:01:59.138861+0000 mon.smithi053 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.146:0/1140770502' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:02:00.555 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:02:00.555 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:02:00.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:00 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:01:59.151359+0000 mgr.smithi053.fgsmol (mgr.14164) 86 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:01.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:01.811+0000 7f897ea39700 1 -- 172.21.15.146:0/3566088686 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89780f6510 msgr2=0x7f89780f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:01.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:01.811+0000 7f897ea39700 1 --2- 172.21.15.146:0/3566088686 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89780f6510 0x7f89780f6930 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f8968009fb0 tx=0x7f8968009b30).stop 2024-01-21T20:02:01.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:01.811+0000 7f897ea39700 1 -- 172.21.15.146:0/3566088686 shutdown_connections 2024-01-21T20:02:01.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:01.811+0000 7f897ea39700 1 --2- 172.21.15.146:0/3566088686 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89780f6510 0x7f89780f6930 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:01.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:01.811+0000 7f897ea39700 1 -- 172.21.15.146:0/3566088686 >> 172.21.15.146:0/3566088686 conn(0x7f89780f1c70 msgr2=0x7f89780f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:01.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:01.811+0000 7f897ea39700 1 -- 172.21.15.146:0/3566088686 shutdown_connections 2024-01-21T20:02:01.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:01.811+0000 7f897ea39700 1 -- 172.21.15.146:0/3566088686 wait complete. 2024-01-21T20:02:01.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:01.812+0000 7f897ea39700 1 Processor -- start 2024-01-21T20:02:01.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:01.812+0000 7f897ea39700 1 -- start start 2024-01-21T20:02:01.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:01.812+0000 7f897ea39700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89780f6510 0x7f8978106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:01.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:01.812+0000 7f897ea39700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8978106c20 con 0x7f89780f6510 2024-01-21T20:02:01.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:01.812+0000 7f8977fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89780f6510 0x7f8978106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:01.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:01.813+0000 7f8977fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89780f6510 0x7f8978106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:56978/0 (socket says 172.21.15.146:56978) 2024-01-21T20:02:01.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:01.813+0000 7f8977fff700 1 -- 172.21.15.146:0/3207972953 learned_addr learned my addr 172.21.15.146:0/3207972953 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:02:01.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:01.813+0000 7f8977fff700 1 -- 172.21.15.146:0/3207972953 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8968005040 con 0x7f89780f6510 2024-01-21T20:02:01.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:01.813+0000 7f8977fff700 1 --2- 172.21.15.146:0/3207972953 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89780f6510 0x7f8978106730 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f8968013510 tx=0x7f896800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:01.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:01.813+0000 7f8974ff9700 1 -- 172.21.15.146:0/3207972953 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8968009320 con 0x7f89780f6510 2024-01-21T20:02:01.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:01.813+0000 7f897ea39700 1 -- 172.21.15.146:0/3207972953 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8978106e20 con 0x7f89780f6510 2024-01-21T20:02:01.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:01.813+0000 7f8974ff9700 1 -- 172.21.15.146:0/3207972953 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8968013f00 con 0x7f89780f6510 2024-01-21T20:02:01.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:01.813+0000 7f8974ff9700 1 -- 172.21.15.146:0/3207972953 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8968014d90 con 0x7f89780f6510 2024-01-21T20:02:01.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:01.813+0000 7f897ea39700 1 -- 172.21.15.146:0/3207972953 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89781072b0 con 0x7f89780f6510 2024-01-21T20:02:01.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:01.814+0000 7f8974ff9700 1 -- 172.21.15.146:0/3207972953 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f89680263a0 con 0x7f89780f6510 2024-01-21T20:02:01.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:01.814+0000 7f897ea39700 1 -- 172.21.15.146:0/3207972953 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8978045ad0 con 0x7f89780f6510 2024-01-21T20:02:01.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:01.814+0000 7f8974ff9700 1 --2- 172.21.15.146:0/3207972953 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f896002ff60 0x7f8960032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:01.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:01.814+0000 7f8974ff9700 1 -- 172.21.15.146:0/3207972953 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8968041320 con 0x7f89780f6510 2024-01-21T20:02:01.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:01.814+0000 7f89777fe700 1 --2- 172.21.15.146:0/3207972953 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f896002ff60 0x7f8960032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:01.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:01.815+0000 7f89777fe700 1 --2- 172.21.15.146:0/3207972953 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f896002ff60 0x7f8960032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f896c0096a0 tx=0x7f896c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:01.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:01.819+0000 7f8974ff9700 1 -- 172.21.15.146:0/3207972953 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f896801ccf0 con 0x7f89780f6510 2024-01-21T20:02:02.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:02.022+0000 7f897ea39700 1 -- 172.21.15.146:0/3207972953 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f89780fbf60 con 0x7f89780f6510 2024-01-21T20:02:02.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:02.023+0000 7f8974ff9700 1 -- 172.21.15.146:0/3207972953 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f896801cb10 con 0x7f89780f6510 2024-01-21T20:02:02.014 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:02:02.014 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:02:02.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:02.024+0000 7f897ea39700 1 -- 172.21.15.146:0/3207972953 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f896002ff60 msgr2=0x7f8960032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:02.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:02.024+0000 7f897ea39700 1 --2- 172.21.15.146:0/3207972953 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f896002ff60 0x7f8960032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f896c0096a0 tx=0x7f896c006b40).stop 2024-01-21T20:02:02.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:02.024+0000 7f897ea39700 1 -- 172.21.15.146:0/3207972953 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89780f6510 msgr2=0x7f8978106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:02.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:02.024+0000 7f897ea39700 1 --2- 172.21.15.146:0/3207972953 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89780f6510 0x7f8978106730 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f8968013510 tx=0x7f896800b040).stop 2024-01-21T20:02:02.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:02.024+0000 7f897ea39700 1 -- 172.21.15.146:0/3207972953 shutdown_connections 2024-01-21T20:02:02.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:02.024+0000 7f897ea39700 1 --2- 172.21.15.146:0/3207972953 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f896002ff60 0x7f8960032420 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:02.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:02.024+0000 7f897ea39700 1 --2- 172.21.15.146:0/3207972953 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89780f6510 0x7f8978106730 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:02.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:02.024+0000 7f897ea39700 1 -- 172.21.15.146:0/3207972953 >> 172.21.15.146:0/3207972953 conn(0x7f89780f1c70 msgr2=0x7f89780f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:02.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:02.025+0000 7f897ea39700 1 -- 172.21.15.146:0/3207972953 shutdown_connections 2024-01-21T20:02:02.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:02.025+0000 7f897ea39700 1 -- 172.21.15.146:0/3207972953 wait complete. 2024-01-21T20:02:02.017 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:02:02.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:02 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:02:01.151729+0000 mgr.smithi053.fgsmol (mgr.14164) 87 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:02.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:02 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:02:02.023960+0000 mon.smithi053 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.146:0/3207972953' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:02:03.415 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:02:03.416 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:02:04.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:04.638+0000 7f40b0a0d700 1 -- 172.21.15.146:0/3795512825 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40ac0f5410 msgr2=0x7f40ac0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:04.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:04.638+0000 7f40b0a0d700 1 --2- 172.21.15.146:0/3795512825 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40ac0f5410 0x7f40ac0f5830 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f4094009fb0 tx=0x7f4094009b30).stop 2024-01-21T20:02:04.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:04.639+0000 7f40b0a0d700 1 -- 172.21.15.146:0/3795512825 shutdown_connections 2024-01-21T20:02:04.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:04.639+0000 7f40b0a0d700 1 --2- 172.21.15.146:0/3795512825 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40ac0f5410 0x7f40ac0f5830 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:04.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:04.639+0000 7f40b0a0d700 1 -- 172.21.15.146:0/3795512825 >> 172.21.15.146:0/3795512825 conn(0x7f40ac0f0b70 msgr2=0x7f40ac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:04.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:04.639+0000 7f40b0a0d700 1 -- 172.21.15.146:0/3795512825 shutdown_connections 2024-01-21T20:02:04.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:04.639+0000 7f40b0a0d700 1 -- 172.21.15.146:0/3795512825 wait complete. 2024-01-21T20:02:04.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:04.639+0000 7f40b0a0d700 1 Processor -- start 2024-01-21T20:02:04.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:04.639+0000 7f40b0a0d700 1 -- start start 2024-01-21T20:02:04.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:04.639+0000 7f40b0a0d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40ac0f5410 0x7f40ac105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:04.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:04.639+0000 7f40b0a0d700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40ac105b20 con 0x7f40ac0f5410 2024-01-21T20:02:04.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:04.639+0000 7f40aa59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40ac0f5410 0x7f40ac105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:04.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:04.639+0000 7f40aa59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40ac0f5410 0x7f40ac105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:56984/0 (socket says 172.21.15.146:56984) 2024-01-21T20:02:04.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:04.639+0000 7f40aa59c700 1 -- 172.21.15.146:0/1493415823 learned_addr learned my addr 172.21.15.146:0/1493415823 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:02:04.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:04.640+0000 7f40aa59c700 1 -- 172.21.15.146:0/1493415823 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4094005040 con 0x7f40ac0f5410 2024-01-21T20:02:04.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:04.640+0000 7f40aa59c700 1 --2- 172.21.15.146:0/1493415823 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40ac0f5410 0x7f40ac105630 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f4094013510 tx=0x7f409400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:04.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:04.640+0000 7f40a2ffd700 1 -- 172.21.15.146:0/1493415823 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4094009320 con 0x7f40ac0f5410 2024-01-21T20:02:04.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:04.640+0000 7f40b0a0d700 1 -- 172.21.15.146:0/1493415823 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40ac105d20 con 0x7f40ac0f5410 2024-01-21T20:02:04.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:04.640+0000 7f40a2ffd700 1 -- 172.21.15.146:0/1493415823 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4094013f00 con 0x7f40ac0f5410 2024-01-21T20:02:04.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:04.640+0000 7f40a2ffd700 1 -- 172.21.15.146:0/1493415823 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4094014d90 con 0x7f40ac0f5410 2024-01-21T20:02:04.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:04.640+0000 7f40b0a0d700 1 -- 172.21.15.146:0/1493415823 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40ac1061b0 con 0x7f40ac0f5410 2024-01-21T20:02:04.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:04.641+0000 7f40a2ffd700 1 -- 172.21.15.146:0/1493415823 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4094014420 con 0x7f40ac0f5410 2024-01-21T20:02:04.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:04.641+0000 7f40b0a0d700 1 -- 172.21.15.146:0/1493415823 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40ac045ad0 con 0x7f40ac0f5410 2024-01-21T20:02:04.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:04.641+0000 7f40a2ffd700 1 --2- 172.21.15.146:0/1493415823 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f409802ff60 0x7f4098032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:04.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:04.641+0000 7f40a2ffd700 1 -- 172.21.15.146:0/1493415823 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4094041070 con 0x7f40ac0f5410 2024-01-21T20:02:04.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:04.641+0000 7f40a9d9b700 1 --2- 172.21.15.146:0/1493415823 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f409802ff60 0x7f4098032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:04.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:04.642+0000 7f40a9d9b700 1 --2- 172.21.15.146:0/1493415823 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f409802ff60 0x7f4098032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f409c0096a0 tx=0x7f409c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:04.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:04.646+0000 7f40a2ffd700 1 -- 172.21.15.146:0/1493415823 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f40940095f0 con 0x7f40ac0f5410 2024-01-21T20:02:04.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:04.851+0000 7f40b0a0d700 1 -- 172.21.15.146:0/1493415823 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f40ac0fae60 con 0x7f40ac0f5410 2024-01-21T20:02:04.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:04.852+0000 7f40a2ffd700 1 -- 172.21.15.146:0/1493415823 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f409401db70 con 0x7f40ac0f5410 2024-01-21T20:02:04.843 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:02:04.843 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:02:04.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:04.853+0000 7f40b0a0d700 1 -- 172.21.15.146:0/1493415823 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f409802ff60 msgr2=0x7f4098032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:04.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:04.853+0000 7f40b0a0d700 1 --2- 172.21.15.146:0/1493415823 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f409802ff60 0x7f4098032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f409c0096a0 tx=0x7f409c006b40).stop 2024-01-21T20:02:04.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:04.853+0000 7f40b0a0d700 1 -- 172.21.15.146:0/1493415823 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40ac0f5410 msgr2=0x7f40ac105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:04.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:04.853+0000 7f40b0a0d700 1 --2- 172.21.15.146:0/1493415823 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40ac0f5410 0x7f40ac105630 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f4094013510 tx=0x7f409400b040).stop 2024-01-21T20:02:04.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:04.854+0000 7f40b0a0d700 1 -- 172.21.15.146:0/1493415823 shutdown_connections 2024-01-21T20:02:04.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:04.854+0000 7f40b0a0d700 1 --2- 172.21.15.146:0/1493415823 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f409802ff60 0x7f4098032420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:04.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:04.854+0000 7f40b0a0d700 1 --2- 172.21.15.146:0/1493415823 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40ac0f5410 0x7f40ac105630 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:04.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:04.854+0000 7f40b0a0d700 1 -- 172.21.15.146:0/1493415823 >> 172.21.15.146:0/1493415823 conn(0x7f40ac0f0b70 msgr2=0x7f40ac0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:04.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:04.854+0000 7f40b0a0d700 1 -- 172.21.15.146:0/1493415823 shutdown_connections 2024-01-21T20:02:04.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:04.854+0000 7f40b0a0d700 1 -- 172.21.15.146:0/1493415823 wait complete. 2024-01-21T20:02:04.846 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:02:04.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:04 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:02:03.152095+0000 mgr.smithi053.fgsmol (mgr.14164) 88 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:05.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:05 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:02:04.853317+0000 mon.smithi053 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.146:0/1493415823' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:02:06.240 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:02:06.241 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:02:06.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:06 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:02:05.152441+0000 mgr.smithi053.fgsmol (mgr.14164) 89 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:07.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:07.522+0000 7f53595cd700 1 -- 172.21.15.146:0/3402364136 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53540f69d0 msgr2=0x7f53540f6df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:07.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:07.522+0000 7f53595cd700 1 --2- 172.21.15.146:0/3402364136 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53540f69d0 0x7f53540f6df0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f5348009fb0 tx=0x7f5348009b30).stop 2024-01-21T20:02:07.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:07.522+0000 7f53595cd700 1 -- 172.21.15.146:0/3402364136 shutdown_connections 2024-01-21T20:02:07.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:07.522+0000 7f53595cd700 1 --2- 172.21.15.146:0/3402364136 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53540f69d0 0x7f53540f6df0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:07.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:07.522+0000 7f53595cd700 1 -- 172.21.15.146:0/3402364136 >> 172.21.15.146:0/3402364136 conn(0x7f53540f02e0 msgr2=0x7f53540f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:07.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:07.522+0000 7f53595cd700 1 -- 172.21.15.146:0/3402364136 shutdown_connections 2024-01-21T20:02:07.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:07.522+0000 7f53595cd700 1 -- 172.21.15.146:0/3402364136 wait complete. 2024-01-21T20:02:07.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:07.523+0000 7f53595cd700 1 Processor -- start 2024-01-21T20:02:07.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:07.523+0000 7f53595cd700 1 -- start start 2024-01-21T20:02:07.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:07.523+0000 7f53595cd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53540f69d0 0x7f53541066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:07.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:07.523+0000 7f53595cd700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5354106bd0 con 0x7f53540f69d0 2024-01-21T20:02:07.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:07.524+0000 7f5352ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53540f69d0 0x7f53541066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:07.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:07.524+0000 7f5352ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53540f69d0 0x7f53541066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:32896/0 (socket says 172.21.15.146:32896) 2024-01-21T20:02:07.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:07.524+0000 7f5352ffd700 1 -- 172.21.15.146:0/4118591446 learned_addr learned my addr 172.21.15.146:0/4118591446 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:02:07.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:07.524+0000 7f5352ffd700 1 -- 172.21.15.146:0/4118591446 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5348005040 con 0x7f53540f69d0 2024-01-21T20:02:07.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:07.524+0000 7f5352ffd700 1 --2- 172.21.15.146:0/4118591446 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53540f69d0 0x7f53541066e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f5348013510 tx=0x7f534800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:07.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:07.524+0000 7f533b7fe700 1 -- 172.21.15.146:0/4118591446 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5348009320 con 0x7f53540f69d0 2024-01-21T20:02:07.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:07.524+0000 7f53595cd700 1 -- 172.21.15.146:0/4118591446 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5354106dd0 con 0x7f53540f69d0 2024-01-21T20:02:07.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:07.524+0000 7f533b7fe700 1 -- 172.21.15.146:0/4118591446 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5348013f00 con 0x7f53540f69d0 2024-01-21T20:02:07.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:07.524+0000 7f533b7fe700 1 -- 172.21.15.146:0/4118591446 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5348014d90 con 0x7f53540f69d0 2024-01-21T20:02:07.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:07.524+0000 7f53595cd700 1 -- 172.21.15.146:0/4118591446 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5354107260 con 0x7f53540f69d0 2024-01-21T20:02:07.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:07.525+0000 7f53595cd700 1 -- 172.21.15.146:0/4118591446 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5354045ad0 con 0x7f53540f69d0 2024-01-21T20:02:07.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:07.525+0000 7f533b7fe700 1 -- 172.21.15.146:0/4118591446 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f53480095f0 con 0x7f53540f69d0 2024-01-21T20:02:07.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:07.525+0000 7f533b7fe700 1 --2- 172.21.15.146:0/4118591446 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f533c02ff60 0x7f533c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:07.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:07.525+0000 7f533b7fe700 1 -- 172.21.15.146:0/4118591446 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5348041000 con 0x7f53540f69d0 2024-01-21T20:02:07.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:07.526+0000 7f53527fc700 1 --2- 172.21.15.146:0/4118591446 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f533c02ff60 0x7f533c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:07.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:07.526+0000 7f53527fc700 1 --2- 172.21.15.146:0/4118591446 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f533c02ff60 0x7f533c032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f53440096a0 tx=0x7f5344006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:07.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:07.530+0000 7f533b7fe700 1 -- 172.21.15.146:0/4118591446 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5348014420 con 0x7f53540f69d0 2024-01-21T20:02:07.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:07.734+0000 7f53595cd700 1 -- 172.21.15.146:0/4118591446 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5354108110 con 0x7f53540f69d0 2024-01-21T20:02:07.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:07.735+0000 7f533b7fe700 1 -- 172.21.15.146:0/4118591446 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f534801d7f0 con 0x7f53540f69d0 2024-01-21T20:02:07.726 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:02:07.727 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:02:07.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:07.736+0000 7f53595cd700 1 -- 172.21.15.146:0/4118591446 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f533c02ff60 msgr2=0x7f533c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:07.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:07.736+0000 7f53595cd700 1 --2- 172.21.15.146:0/4118591446 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f533c02ff60 0x7f533c032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f53440096a0 tx=0x7f5344006b40).stop 2024-01-21T20:02:07.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:07.736+0000 7f53595cd700 1 -- 172.21.15.146:0/4118591446 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53540f69d0 msgr2=0x7f53541066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:07.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:07.736+0000 7f53595cd700 1 --2- 172.21.15.146:0/4118591446 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53540f69d0 0x7f53541066e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f5348013510 tx=0x7f534800b040).stop 2024-01-21T20:02:07.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:07.736+0000 7f53595cd700 1 -- 172.21.15.146:0/4118591446 shutdown_connections 2024-01-21T20:02:07.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:07.736+0000 7f53595cd700 1 --2- 172.21.15.146:0/4118591446 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f533c02ff60 0x7f533c032420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:07.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:07.736+0000 7f53595cd700 1 --2- 172.21.15.146:0/4118591446 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53540f69d0 0x7f53541066e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:07.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:07.737+0000 7f53595cd700 1 -- 172.21.15.146:0/4118591446 >> 172.21.15.146:0/4118591446 conn(0x7f53540f02e0 msgr2=0x7f53540f4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:07.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:07.737+0000 7f53595cd700 1 -- 172.21.15.146:0/4118591446 shutdown_connections 2024-01-21T20:02:07.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:07.737+0000 7f53595cd700 1 -- 172.21.15.146:0/4118591446 wait complete. 2024-01-21T20:02:07.729 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:02:08.936 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:08 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:02:07.152816+0000 mgr.smithi053.fgsmol (mgr.14164) 90 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:08.937 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:08 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:02:07.735974+0000 mon.smithi053 (mon.0) 173 : audit [DBG] from='client.? 172.21.15.146:0/4118591446' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:02:09.118 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:02:09.118 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:02:09.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:09 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:02:09.264885+0000 mon.smithi053 (mon.0) 174 : audit [INF] from='mgr.14164 172.21.15.53:0/2331671680' entity='mgr.smithi053.fgsmol' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fgsmol/mirror_snapshot_schedule"}]: dispatch 2024-01-21T20:02:09.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:09 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:02:09.273766+0000 mon.smithi053 (mon.0) 175 : audit [INF] from='mgr.14164 172.21.15.53:0/2331671680' entity='mgr.smithi053.fgsmol' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fgsmol/trash_purge_schedule"}]: dispatch 2024-01-21T20:02:10.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:10.309+0000 7f1208349700 1 -- 172.21.15.146:0/3586458254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12000f6510 msgr2=0x7f12000f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:10.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:10.309+0000 7f1208349700 1 --2- 172.21.15.146:0/3586458254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12000f6510 0x7f12000f6930 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f11fc009fb0 tx=0x7f11fc009b30).stop 2024-01-21T20:02:10.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:10.309+0000 7f1208349700 1 -- 172.21.15.146:0/3586458254 shutdown_connections 2024-01-21T20:02:10.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:10.309+0000 7f1208349700 1 --2- 172.21.15.146:0/3586458254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12000f6510 0x7f12000f6930 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:10.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:10.309+0000 7f1208349700 1 -- 172.21.15.146:0/3586458254 >> 172.21.15.146:0/3586458254 conn(0x7f12000f1c70 msgr2=0x7f12000f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:10.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:10.309+0000 7f1208349700 1 -- 172.21.15.146:0/3586458254 shutdown_connections 2024-01-21T20:02:10.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:10.310+0000 7f1208349700 1 -- 172.21.15.146:0/3586458254 wait complete. 2024-01-21T20:02:10.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:10.310+0000 7f1208349700 1 Processor -- start 2024-01-21T20:02:10.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:10.310+0000 7f1208349700 1 -- start start 2024-01-21T20:02:10.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:10.310+0000 7f1208349700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12000f6510 0x7f1200106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:10.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:10.310+0000 7f1208349700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1200106c20 con 0x7f12000f6510 2024-01-21T20:02:10.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:10.311+0000 7f12060e5700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12000f6510 0x7f1200106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:10.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:10.311+0000 7f12060e5700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12000f6510 0x7f1200106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:32912/0 (socket says 172.21.15.146:32912) 2024-01-21T20:02:10.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:10.311+0000 7f12060e5700 1 -- 172.21.15.146:0/2140296234 learned_addr learned my addr 172.21.15.146:0/2140296234 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:02:10.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:10.311+0000 7f12060e5700 1 -- 172.21.15.146:0/2140296234 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11fc005040 con 0x7f12000f6510 2024-01-21T20:02:10.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:10.311+0000 7f12060e5700 1 --2- 172.21.15.146:0/2140296234 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12000f6510 0x7f1200106730 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f11fc014000 tx=0x7f11fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:10.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:10.311+0000 7f11f6ffd700 1 -- 172.21.15.146:0/2140296234 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11fc0095f0 con 0x7f12000f6510 2024-01-21T20:02:10.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:10.311+0000 7f11f6ffd700 1 -- 172.21.15.146:0/2140296234 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f11fc007e70 con 0x7f12000f6510 2024-01-21T20:02:10.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:10.311+0000 7f1208349700 1 -- 172.21.15.146:0/2140296234 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1200106e20 con 0x7f12000f6510 2024-01-21T20:02:10.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:10.311+0000 7f11f6ffd700 1 -- 172.21.15.146:0/2140296234 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11fc014ea0 con 0x7f12000f6510 2024-01-21T20:02:10.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:10.311+0000 7f1208349700 1 -- 172.21.15.146:0/2140296234 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12001072b0 con 0x7f12000f6510 2024-01-21T20:02:10.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:10.312+0000 7f11f6ffd700 1 -- 172.21.15.146:0/2140296234 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f11fc025070 con 0x7f12000f6510 2024-01-21T20:02:10.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:10.312+0000 7f1208349700 1 -- 172.21.15.146:0/2140296234 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1200045ad0 con 0x7f12000f6510 2024-01-21T20:02:10.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:10.312+0000 7f11f6ffd700 1 --2- 172.21.15.146:0/2140296234 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f11ec02ffb0 0x7f11ec032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:10.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:10.312+0000 7f11f6ffd700 1 -- 172.21.15.146:0/2140296234 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f11fc040930 con 0x7f12000f6510 2024-01-21T20:02:10.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:10.313+0000 7f12058e4700 1 --2- 172.21.15.146:0/2140296234 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f11ec02ffb0 0x7f11ec032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:10.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:10.313+0000 7f12058e4700 1 --2- 172.21.15.146:0/2140296234 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f11ec02ffb0 0x7f11ec032470 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f11f00096a0 tx=0x7f11f0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:10.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:10.317+0000 7f11f6ffd700 1 -- 172.21.15.146:0/2140296234 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f11fc00f4e0 con 0x7f12000f6510 2024-01-21T20:02:10.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:10.524+0000 7f1208349700 1 -- 172.21.15.146:0/2140296234 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f12000fbf60 con 0x7f12000f6510 2024-01-21T20:02:10.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:10.524+0000 7f11f6ffd700 1 -- 172.21.15.146:0/2140296234 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f11fc00f300 con 0x7f12000f6510 2024-01-21T20:02:10.516 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:02:10.516 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:02:10.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:10.526+0000 7f1208349700 1 -- 172.21.15.146:0/2140296234 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f11ec02ffb0 msgr2=0x7f11ec032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:10.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:10.526+0000 7f1208349700 1 --2- 172.21.15.146:0/2140296234 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f11ec02ffb0 0x7f11ec032470 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f11f00096a0 tx=0x7f11f0006b40).stop 2024-01-21T20:02:10.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:10.526+0000 7f1208349700 1 -- 172.21.15.146:0/2140296234 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12000f6510 msgr2=0x7f1200106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:10.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:10.526+0000 7f1208349700 1 --2- 172.21.15.146:0/2140296234 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12000f6510 0x7f1200106730 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f11fc014000 tx=0x7f11fc00b040).stop 2024-01-21T20:02:10.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:10.526+0000 7f1208349700 1 -- 172.21.15.146:0/2140296234 shutdown_connections 2024-01-21T20:02:10.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:10.526+0000 7f1208349700 1 --2- 172.21.15.146:0/2140296234 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f11ec02ffb0 0x7f11ec032470 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:10.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:10.526+0000 7f1208349700 1 --2- 172.21.15.146:0/2140296234 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12000f6510 0x7f1200106730 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:10.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:10.526+0000 7f1208349700 1 -- 172.21.15.146:0/2140296234 >> 172.21.15.146:0/2140296234 conn(0x7f12000f1c70 msgr2=0x7f12000f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:10.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:10.526+0000 7f1208349700 1 -- 172.21.15.146:0/2140296234 shutdown_connections 2024-01-21T20:02:10.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:10.526+0000 7f1208349700 1 -- 172.21.15.146:0/2140296234 wait complete. 2024-01-21T20:02:10.518 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:02:10.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:10 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:02:09.153136+0000 mgr.smithi053.fgsmol (mgr.14164) 91 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:10.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:10 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:02:10.525721+0000 mon.smithi053 (mon.0) 176 : audit [DBG] from='client.? 172.21.15.146:0/2140296234' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:02:11.945 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:02:11.945 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:02:12.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:12 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:02:11.153513+0000 mgr.smithi053.fgsmol (mgr.14164) 92 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:13.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:13.197+0000 7f0655cf0700 1 -- 172.21.15.146:0/1909803894 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0650060220 msgr2=0x7f06500f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:13.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:13.197+0000 7f0655cf0700 1 --2- 172.21.15.146:0/1909803894 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0650060220 0x7f06500f9150 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f0638009fb0 tx=0x7f0638009b30).stop 2024-01-21T20:02:13.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:13.197+0000 7f0655cf0700 1 -- 172.21.15.146:0/1909803894 shutdown_connections 2024-01-21T20:02:13.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:13.197+0000 7f0655cf0700 1 --2- 172.21.15.146:0/1909803894 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0650060220 0x7f06500f9150 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:13.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:13.197+0000 7f0655cf0700 1 -- 172.21.15.146:0/1909803894 >> 172.21.15.146:0/1909803894 conn(0x7f06500f0b70 msgr2=0x7f06500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:13.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:13.197+0000 7f0655cf0700 1 -- 172.21.15.146:0/1909803894 shutdown_connections 2024-01-21T20:02:13.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:13.197+0000 7f0655cf0700 1 -- 172.21.15.146:0/1909803894 wait complete. 2024-01-21T20:02:13.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:13.198+0000 7f0655cf0700 1 Processor -- start 2024-01-21T20:02:13.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:13.198+0000 7f0655cf0700 1 -- start start 2024-01-21T20:02:13.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:13.198+0000 7f0655cf0700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0650060220 0x7f06500f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:13.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:13.198+0000 7f0655cf0700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06500f5050 con 0x7f0650060220 2024-01-21T20:02:13.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:13.198+0000 7f064f7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0650060220 0x7f06500f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:13.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:13.198+0000 7f064f7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0650060220 0x7f06500f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:32926/0 (socket says 172.21.15.146:32926) 2024-01-21T20:02:13.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:13.198+0000 7f064f7fe700 1 -- 172.21.15.146:0/2981530046 learned_addr learned my addr 172.21.15.146:0/2981530046 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:02:13.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:13.199+0000 7f064f7fe700 1 -- 172.21.15.146:0/2981530046 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0638005040 con 0x7f0650060220 2024-01-21T20:02:13.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:13.199+0000 7f064f7fe700 1 --2- 172.21.15.146:0/2981530046 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0650060220 0x7f06500f6bd0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f0638013510 tx=0x7f063800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:13.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:13.199+0000 7f0647fff700 1 -- 172.21.15.146:0/2981530046 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0638009320 con 0x7f0650060220 2024-01-21T20:02:13.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:13.199+0000 7f0647fff700 1 -- 172.21.15.146:0/2981530046 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0638013f00 con 0x7f0650060220 2024-01-21T20:02:13.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:13.199+0000 7f0655cf0700 1 -- 172.21.15.146:0/2981530046 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06500f51f0 con 0x7f0650060220 2024-01-21T20:02:13.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:13.199+0000 7f0647fff700 1 -- 172.21.15.146:0/2981530046 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0638014dc0 con 0x7f0650060220 2024-01-21T20:02:13.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:13.199+0000 7f0655cf0700 1 -- 172.21.15.146:0/2981530046 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06500f5680 con 0x7f0650060220 2024-01-21T20:02:13.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:13.200+0000 7f0647fff700 1 -- 172.21.15.146:0/2981530046 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0638014420 con 0x7f0650060220 2024-01-21T20:02:13.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:13.200+0000 7f0655cf0700 1 -- 172.21.15.146:0/2981530046 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0650059100 con 0x7f0650060220 2024-01-21T20:02:13.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:13.200+0000 7f0647fff700 1 --2- 172.21.15.146:0/2981530046 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f063c030000 0x7f063c0324c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:13.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:13.200+0000 7f0647fff700 1 -- 172.21.15.146:0/2981530046 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f06380410b0 con 0x7f0650060220 2024-01-21T20:02:13.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:13.200+0000 7f064effd700 1 --2- 172.21.15.146:0/2981530046 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f063c030000 0x7f063c0324c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:13.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:13.201+0000 7f064effd700 1 --2- 172.21.15.146:0/2981530046 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f063c030000 0x7f063c0324c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f06400096a0 tx=0x7f0640006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:13.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:13.205+0000 7f0647fff700 1 -- 172.21.15.146:0/2981530046 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f06380095f0 con 0x7f0650060220 2024-01-21T20:02:13.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:13.410+0000 7f0655cf0700 1 -- 172.21.15.146:0/2981530046 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f06500fbf50 con 0x7f0650060220 2024-01-21T20:02:13.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:13.410+0000 7f0647fff700 1 -- 172.21.15.146:0/2981530046 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f063801db70 con 0x7f0650060220 2024-01-21T20:02:13.401 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:02:13.402 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:02:13.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:13.412+0000 7f0655cf0700 1 -- 172.21.15.146:0/2981530046 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f063c030000 msgr2=0x7f063c0324c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:13.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:13.412+0000 7f0655cf0700 1 --2- 172.21.15.146:0/2981530046 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f063c030000 0x7f063c0324c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f06400096a0 tx=0x7f0640006b40).stop 2024-01-21T20:02:13.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:13.412+0000 7f0655cf0700 1 -- 172.21.15.146:0/2981530046 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0650060220 msgr2=0x7f06500f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:13.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:13.412+0000 7f0655cf0700 1 --2- 172.21.15.146:0/2981530046 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0650060220 0x7f06500f6bd0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f0638013510 tx=0x7f063800b040).stop 2024-01-21T20:02:13.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:13.412+0000 7f0655cf0700 1 -- 172.21.15.146:0/2981530046 shutdown_connections 2024-01-21T20:02:13.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:13.412+0000 7f0655cf0700 1 --2- 172.21.15.146:0/2981530046 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f063c030000 0x7f063c0324c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:13.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:13.412+0000 7f0655cf0700 1 --2- 172.21.15.146:0/2981530046 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0650060220 0x7f06500f6bd0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:13.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:13.412+0000 7f0655cf0700 1 -- 172.21.15.146:0/2981530046 >> 172.21.15.146:0/2981530046 conn(0x7f06500f0b70 msgr2=0x7f06500f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:13.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:13.412+0000 7f0655cf0700 1 -- 172.21.15.146:0/2981530046 shutdown_connections 2024-01-21T20:02:13.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:13.412+0000 7f0655cf0700 1 -- 172.21.15.146:0/2981530046 wait complete. 2024-01-21T20:02:13.404 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:02:13.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:13 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:02:13.411458+0000 mon.smithi053 (mon.0) 177 : audit [DBG] from='client.? 172.21.15.146:0/2981530046' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:02:14.871 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:02:14.871 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:02:14.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:14 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:02:13.153925+0000 mgr.smithi053.fgsmol (mgr.14164) 93 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:16.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:16.061+0000 7f7b42248700 1 -- 172.21.15.146:0/4012165611 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b3c060220 msgr2=0x7f7b3c0f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:16.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:16.061+0000 7f7b42248700 1 --2- 172.21.15.146:0/4012165611 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b3c060220 0x7f7b3c0f9150 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f7b2c009fb0 tx=0x7f7b2c009b30).stop 2024-01-21T20:02:16.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:16.061+0000 7f7b42248700 1 -- 172.21.15.146:0/4012165611 shutdown_connections 2024-01-21T20:02:16.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:16.061+0000 7f7b42248700 1 --2- 172.21.15.146:0/4012165611 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b3c060220 0x7f7b3c0f9150 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:16.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:16.061+0000 7f7b42248700 1 -- 172.21.15.146:0/4012165611 >> 172.21.15.146:0/4012165611 conn(0x7f7b3c0f0b70 msgr2=0x7f7b3c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:16.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:16.061+0000 7f7b42248700 1 -- 172.21.15.146:0/4012165611 shutdown_connections 2024-01-21T20:02:16.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:16.061+0000 7f7b42248700 1 -- 172.21.15.146:0/4012165611 wait complete. 2024-01-21T20:02:16.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:16.062+0000 7f7b42248700 1 Processor -- start 2024-01-21T20:02:16.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:16.062+0000 7f7b42248700 1 -- start start 2024-01-21T20:02:16.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:16.062+0000 7f7b42248700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b3c060220 0x7f7b3c1066a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:16.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:16.062+0000 7f7b42248700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3c106b90 con 0x7f7b3c060220 2024-01-21T20:02:16.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:16.063+0000 7f7b3b7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b3c060220 0x7f7b3c1066a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:16.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:16.063+0000 7f7b3b7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b3c060220 0x7f7b3c1066a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:32948/0 (socket says 172.21.15.146:32948) 2024-01-21T20:02:16.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:16.063+0000 7f7b3b7fe700 1 -- 172.21.15.146:0/2527730127 learned_addr learned my addr 172.21.15.146:0/2527730127 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:02:16.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:16.063+0000 7f7b3b7fe700 1 -- 172.21.15.146:0/2527730127 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b2c005040 con 0x7f7b3c060220 2024-01-21T20:02:16.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:16.063+0000 7f7b3b7fe700 1 --2- 172.21.15.146:0/2527730127 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b3c060220 0x7f7b3c1066a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f7b2c013510 tx=0x7f7b2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:16.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:16.063+0000 7f7b23fff700 1 -- 172.21.15.146:0/2527730127 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b2c009320 con 0x7f7b3c060220 2024-01-21T20:02:16.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:16.063+0000 7f7b42248700 1 -- 172.21.15.146:0/2527730127 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b3c106d90 con 0x7f7b3c060220 2024-01-21T20:02:16.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:16.063+0000 7f7b23fff700 1 -- 172.21.15.146:0/2527730127 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7b2c013f00 con 0x7f7b3c060220 2024-01-21T20:02:16.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:16.063+0000 7f7b23fff700 1 -- 172.21.15.146:0/2527730127 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b2c014dc0 con 0x7f7b3c060220 2024-01-21T20:02:16.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:16.063+0000 7f7b42248700 1 -- 172.21.15.146:0/2527730127 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b3c107220 con 0x7f7b3c060220 2024-01-21T20:02:16.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:16.064+0000 7f7b23fff700 1 -- 172.21.15.146:0/2527730127 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7b2c0095f0 con 0x7f7b3c060220 2024-01-21T20:02:16.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:16.064+0000 7f7b42248700 1 -- 172.21.15.146:0/2527730127 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b3c045ad0 con 0x7f7b3c060220 2024-01-21T20:02:16.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:16.064+0000 7f7b23fff700 1 --2- 172.21.15.146:0/2527730127 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f7b2402ffb0 0x7f7b24032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:16.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:16.065+0000 7f7b23fff700 1 -- 172.21.15.146:0/2527730127 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7b2c0410c0 con 0x7f7b3c060220 2024-01-21T20:02:16.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:16.065+0000 7f7b3affd700 1 --2- 172.21.15.146:0/2527730127 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f7b2402ffb0 0x7f7b24032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:16.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:16.065+0000 7f7b3affd700 1 --2- 172.21.15.146:0/2527730127 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f7b2402ffb0 0x7f7b24032470 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f7b300096a0 tx=0x7f7b30006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:16.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:16.069+0000 7f7b23fff700 1 -- 172.21.15.146:0/2527730127 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7b2c014420 con 0x7f7b3c060220 2024-01-21T20:02:16.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:16.274+0000 7f7b42248700 1 -- 172.21.15.146:0/2527730127 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7b3c1080d0 con 0x7f7b3c060220 2024-01-21T20:02:16.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:16.275+0000 7f7b23fff700 1 -- 172.21.15.146:0/2527730127 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7b2c010230 con 0x7f7b3c060220 2024-01-21T20:02:16.266 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:02:16.267 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:02:16.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:16.277+0000 7f7b42248700 1 -- 172.21.15.146:0/2527730127 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f7b2402ffb0 msgr2=0x7f7b24032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:16.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:16.277+0000 7f7b42248700 1 --2- 172.21.15.146:0/2527730127 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f7b2402ffb0 0x7f7b24032470 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f7b300096a0 tx=0x7f7b30006b40).stop 2024-01-21T20:02:16.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:16.277+0000 7f7b42248700 1 -- 172.21.15.146:0/2527730127 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b3c060220 msgr2=0x7f7b3c1066a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:16.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:16.277+0000 7f7b42248700 1 --2- 172.21.15.146:0/2527730127 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b3c060220 0x7f7b3c1066a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f7b2c013510 tx=0x7f7b2c00b040).stop 2024-01-21T20:02:16.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:16.277+0000 7f7b42248700 1 -- 172.21.15.146:0/2527730127 shutdown_connections 2024-01-21T20:02:16.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:16.277+0000 7f7b42248700 1 --2- 172.21.15.146:0/2527730127 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f7b2402ffb0 0x7f7b24032470 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:16.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:16.277+0000 7f7b42248700 1 --2- 172.21.15.146:0/2527730127 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b3c060220 0x7f7b3c1066a0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:16.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:16.277+0000 7f7b42248700 1 -- 172.21.15.146:0/2527730127 >> 172.21.15.146:0/2527730127 conn(0x7f7b3c0f0b70 msgr2=0x7f7b3c0f16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:16.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:16.277+0000 7f7b42248700 1 -- 172.21.15.146:0/2527730127 shutdown_connections 2024-01-21T20:02:16.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:16.277+0000 7f7b42248700 1 -- 172.21.15.146:0/2527730127 wait complete. 2024-01-21T20:02:16.269 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:02:16.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:02:15.154382+0000 mgr.smithi053.fgsmol (mgr.14164) 94 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:16.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:02:16.276510+0000 mon.smithi053 (mon.0) 178 : audit [DBG] from='client.? 172.21.15.146:0/2527730127' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:02:17.687 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:02:17.687 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:02:18.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:18.937+0000 7f4de16ab700 1 -- 172.21.15.146:0/28104011 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ddc0f5410 msgr2=0x7f4ddc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:18.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:18.937+0000 7f4de16ab700 1 --2- 172.21.15.146:0/28104011 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ddc0f5410 0x7f4ddc0f5830 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f4dd0009fb0 tx=0x7f4dd0009b30).stop 2024-01-21T20:02:18.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:18.937+0000 7f4de16ab700 1 -- 172.21.15.146:0/28104011 shutdown_connections 2024-01-21T20:02:18.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:18.937+0000 7f4de16ab700 1 --2- 172.21.15.146:0/28104011 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ddc0f5410 0x7f4ddc0f5830 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:18.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:18.937+0000 7f4de16ab700 1 -- 172.21.15.146:0/28104011 >> 172.21.15.146:0/28104011 conn(0x7f4ddc0f0b70 msgr2=0x7f4ddc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:18.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:18.937+0000 7f4de16ab700 1 -- 172.21.15.146:0/28104011 shutdown_connections 2024-01-21T20:02:18.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:18.937+0000 7f4de16ab700 1 -- 172.21.15.146:0/28104011 wait complete. 2024-01-21T20:02:18.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:18.938+0000 7f4de16ab700 1 Processor -- start 2024-01-21T20:02:18.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:18.938+0000 7f4de16ab700 1 -- start start 2024-01-21T20:02:18.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:18.938+0000 7f4de16ab700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ddc0f5410 0x7f4ddc109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:18.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:18.938+0000 7f4de16ab700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ddc060220 con 0x7f4ddc0f5410 2024-01-21T20:02:18.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:18.938+0000 7f4ddaffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ddc0f5410 0x7f4ddc109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:18.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:18.938+0000 7f4ddaffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ddc0f5410 0x7f4ddc109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:53820/0 (socket says 172.21.15.146:53820) 2024-01-21T20:02:18.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:18.938+0000 7f4ddaffd700 1 -- 172.21.15.146:0/3191546950 learned_addr learned my addr 172.21.15.146:0/3191546950 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:02:18.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:18.939+0000 7f4ddaffd700 1 -- 172.21.15.146:0/3191546950 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd0005040 con 0x7f4ddc0f5410 2024-01-21T20:02:18.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:18.939+0000 7f4ddaffd700 1 --2- 172.21.15.146:0/3191546950 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ddc0f5410 0x7f4ddc109450 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f4dd0013510 tx=0x7f4dd000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:18.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:18.939+0000 7f4dc37fe700 1 -- 172.21.15.146:0/3191546950 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4dd0009320 con 0x7f4ddc0f5410 2024-01-21T20:02:18.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:18.939+0000 7f4de16ab700 1 -- 172.21.15.146:0/3191546950 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ddc060420 con 0x7f4ddc0f5410 2024-01-21T20:02:18.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:18.939+0000 7f4dc37fe700 1 -- 172.21.15.146:0/3191546950 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4dd0013f00 con 0x7f4ddc0f5410 2024-01-21T20:02:18.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:18.939+0000 7f4dc37fe700 1 -- 172.21.15.146:0/3191546950 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4dd0014d90 con 0x7f4ddc0f5410 2024-01-21T20:02:18.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:18.939+0000 7f4de16ab700 1 -- 172.21.15.146:0/3191546950 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ddc109b00 con 0x7f4ddc0f5410 2024-01-21T20:02:18.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:18.940+0000 7f4dc37fe700 1 -- 172.21.15.146:0/3191546950 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4dd00263a0 con 0x7f4ddc0f5410 2024-01-21T20:02:18.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:18.940+0000 7f4de16ab700 1 -- 172.21.15.146:0/3191546950 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ddc059100 con 0x7f4ddc0f5410 2024-01-21T20:02:18.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:18.940+0000 7f4dc37fe700 1 --2- 172.21.15.146:0/3191546950 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f4dc402ff60 0x7f4dc4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:18.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:18.940+0000 7f4dc37fe700 1 -- 172.21.15.146:0/3191546950 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4dd0041100 con 0x7f4ddc0f5410 2024-01-21T20:02:18.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:18.940+0000 7f4dda7fc700 1 --2- 172.21.15.146:0/3191546950 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f4dc402ff60 0x7f4dc4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:18.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:18.941+0000 7f4dda7fc700 1 --2- 172.21.15.146:0/3191546950 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f4dc402ff60 0x7f4dc4032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f4dcc0096a0 tx=0x7f4dcc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:18.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:18 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:02:17.154784+0000 mgr.smithi053.fgsmol (mgr.14164) 95 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:18.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:18.945+0000 7f4dc37fe700 1 -- 172.21.15.146:0/3191546950 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4dd00095f0 con 0x7f4ddc0f5410 2024-01-21T20:02:19.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:19.147+0000 7f4de16ab700 1 -- 172.21.15.146:0/3191546950 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4ddc045ad0 con 0x7f4ddc0f5410 2024-01-21T20:02:19.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:19.148+0000 7f4dc37fe700 1 -- 172.21.15.146:0/3191546950 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4dd001db70 con 0x7f4ddc0f5410 2024-01-21T20:02:19.139 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:02:19.139 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:02:19.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:19.149+0000 7f4de16ab700 1 -- 172.21.15.146:0/3191546950 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f4dc402ff60 msgr2=0x7f4dc4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:19.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:19.149+0000 7f4de16ab700 1 --2- 172.21.15.146:0/3191546950 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f4dc402ff60 0x7f4dc4032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f4dcc0096a0 tx=0x7f4dcc006b40).stop 2024-01-21T20:02:19.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:19.149+0000 7f4de16ab700 1 -- 172.21.15.146:0/3191546950 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ddc0f5410 msgr2=0x7f4ddc109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:19.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:19.149+0000 7f4de16ab700 1 --2- 172.21.15.146:0/3191546950 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ddc0f5410 0x7f4ddc109450 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f4dd0013510 tx=0x7f4dd000b040).stop 2024-01-21T20:02:19.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:19.149+0000 7f4de16ab700 1 -- 172.21.15.146:0/3191546950 shutdown_connections 2024-01-21T20:02:19.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:19.149+0000 7f4de16ab700 1 --2- 172.21.15.146:0/3191546950 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f4dc402ff60 0x7f4dc4032420 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:19.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:19.149+0000 7f4de16ab700 1 --2- 172.21.15.146:0/3191546950 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ddc0f5410 0x7f4ddc109450 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:19.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:19.149+0000 7f4de16ab700 1 -- 172.21.15.146:0/3191546950 >> 172.21.15.146:0/3191546950 conn(0x7f4ddc0f0b70 msgr2=0x7f4ddc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:19.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:19.149+0000 7f4de16ab700 1 -- 172.21.15.146:0/3191546950 shutdown_connections 2024-01-21T20:02:19.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:19.149+0000 7f4de16ab700 1 -- 172.21.15.146:0/3191546950 wait complete. 2024-01-21T20:02:19.142 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:02:19.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:19 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:02:19.149006+0000 mon.smithi053 (mon.0) 179 : audit [DBG] from='client.? 172.21.15.146:0/3191546950' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:02:20.603 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:02:20.604 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:02:20.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:20 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:02:19.155299+0000 mgr.smithi053.fgsmol (mgr.14164) 96 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:21.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:21.798+0000 7f1cbfd5c700 1 -- 172.21.15.146:0/2892615611 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cb80f5410 msgr2=0x7f1cb80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:21.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:21.798+0000 7f1cbfd5c700 1 --2- 172.21.15.146:0/2892615611 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cb80f5410 0x7f1cb80f5830 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f1cb4009fb0 tx=0x7f1cb4009b30).stop 2024-01-21T20:02:21.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:21.798+0000 7f1cbfd5c700 1 -- 172.21.15.146:0/2892615611 shutdown_connections 2024-01-21T20:02:21.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:21.798+0000 7f1cbfd5c700 1 --2- 172.21.15.146:0/2892615611 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cb80f5410 0x7f1cb80f5830 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:21.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:21.798+0000 7f1cbfd5c700 1 -- 172.21.15.146:0/2892615611 >> 172.21.15.146:0/2892615611 conn(0x7f1cb80f0b70 msgr2=0x7f1cb80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:21.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:21.798+0000 7f1cbfd5c700 1 -- 172.21.15.146:0/2892615611 shutdown_connections 2024-01-21T20:02:21.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:21.798+0000 7f1cbfd5c700 1 -- 172.21.15.146:0/2892615611 wait complete. 2024-01-21T20:02:21.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:21.799+0000 7f1cbfd5c700 1 Processor -- start 2024-01-21T20:02:21.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:21.799+0000 7f1cbfd5c700 1 -- start start 2024-01-21T20:02:21.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:21.799+0000 7f1cbfd5c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cb80f5410 0x7f1cb8109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:21.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:21.799+0000 7f1cbfd5c700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb8060240 con 0x7f1cb80f5410 2024-01-21T20:02:21.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:21.799+0000 7f1cbdaf8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cb80f5410 0x7f1cb8109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:21.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:21.800+0000 7f1cbdaf8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cb80f5410 0x7f1cb8109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:53836/0 (socket says 172.21.15.146:53836) 2024-01-21T20:02:21.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:21.800+0000 7f1cbdaf8700 1 -- 172.21.15.146:0/2485848463 learned_addr learned my addr 172.21.15.146:0/2485848463 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:02:21.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:21.800+0000 7f1cbdaf8700 1 -- 172.21.15.146:0/2485848463 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cb4005040 con 0x7f1cb80f5410 2024-01-21T20:02:21.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:21.800+0000 7f1cbdaf8700 1 --2- 172.21.15.146:0/2485848463 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cb80f5410 0x7f1cb8109430 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f1cb4013510 tx=0x7f1cb400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:21.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:21.800+0000 7f1cae7fc700 1 -- 172.21.15.146:0/2485848463 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1cb4009320 con 0x7f1cb80f5410 2024-01-21T20:02:21.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:21.800+0000 7f1cbfd5c700 1 -- 172.21.15.146:0/2485848463 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cb8060440 con 0x7f1cb80f5410 2024-01-21T20:02:21.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:21.800+0000 7f1cae7fc700 1 -- 172.21.15.146:0/2485848463 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1cb4013f00 con 0x7f1cb80f5410 2024-01-21T20:02:21.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:21.800+0000 7f1cae7fc700 1 -- 172.21.15.146:0/2485848463 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1cb4014d90 con 0x7f1cb80f5410 2024-01-21T20:02:21.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:21.800+0000 7f1cbfd5c700 1 -- 172.21.15.146:0/2485848463 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cb8109ae0 con 0x7f1cb80f5410 2024-01-21T20:02:21.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:21.801+0000 7f1cae7fc700 1 -- 172.21.15.146:0/2485848463 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1cb4014420 con 0x7f1cb80f5410 2024-01-21T20:02:21.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:21.801+0000 7f1cbfd5c700 1 -- 172.21.15.146:0/2485848463 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cb8045ad0 con 0x7f1cb80f5410 2024-01-21T20:02:21.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:21.801+0000 7f1cae7fc700 1 --2- 172.21.15.146:0/2485848463 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f1ca402ff60 0x7f1ca4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:21.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:21.801+0000 7f1cae7fc700 1 -- 172.21.15.146:0/2485848463 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1cb4041000 con 0x7f1cb80f5410 2024-01-21T20:02:21.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:21.801+0000 7f1cbd2f7700 1 --2- 172.21.15.146:0/2485848463 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f1ca402ff60 0x7f1ca4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:21.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:21.802+0000 7f1cbd2f7700 1 --2- 172.21.15.146:0/2485848463 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f1ca402ff60 0x7f1ca4032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f1ca80096a0 tx=0x7f1ca8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:21.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:21.806+0000 7f1cae7fc700 1 -- 172.21.15.146:0/2485848463 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1cb40095f0 con 0x7f1cb80f5410 2024-01-21T20:02:22.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:22.013+0000 7f1cbfd5c700 1 -- 172.21.15.146:0/2485848463 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1cb80fae60 con 0x7f1cb80f5410 2024-01-21T20:02:22.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:22.013+0000 7f1cae7fc700 1 -- 172.21.15.146:0/2485848463 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1cb401db70 con 0x7f1cb80f5410 2024-01-21T20:02:22.005 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:02:22.005 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:02:22.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:22.015+0000 7f1cbfd5c700 1 -- 172.21.15.146:0/2485848463 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f1ca402ff60 msgr2=0x7f1ca4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:22.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:22.015+0000 7f1cbfd5c700 1 --2- 172.21.15.146:0/2485848463 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f1ca402ff60 0x7f1ca4032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f1ca80096a0 tx=0x7f1ca8006b40).stop 2024-01-21T20:02:22.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:22.015+0000 7f1cbfd5c700 1 -- 172.21.15.146:0/2485848463 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cb80f5410 msgr2=0x7f1cb8109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:22.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:22.015+0000 7f1cbfd5c700 1 --2- 172.21.15.146:0/2485848463 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cb80f5410 0x7f1cb8109430 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f1cb4013510 tx=0x7f1cb400b040).stop 2024-01-21T20:02:22.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:22.015+0000 7f1cbfd5c700 1 -- 172.21.15.146:0/2485848463 shutdown_connections 2024-01-21T20:02:22.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:22.015+0000 7f1cbfd5c700 1 --2- 172.21.15.146:0/2485848463 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f1ca402ff60 0x7f1ca4032420 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:22.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:22.015+0000 7f1cbfd5c700 1 --2- 172.21.15.146:0/2485848463 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cb80f5410 0x7f1cb8109430 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:22.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:22.015+0000 7f1cbfd5c700 1 -- 172.21.15.146:0/2485848463 >> 172.21.15.146:0/2485848463 conn(0x7f1cb80f0b70 msgr2=0x7f1cb80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:22.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:22.015+0000 7f1cbfd5c700 1 -- 172.21.15.146:0/2485848463 shutdown_connections 2024-01-21T20:02:22.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:22.015+0000 7f1cbfd5c700 1 -- 172.21.15.146:0/2485848463 wait complete. 2024-01-21T20:02:22.007 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:02:22.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:02:21.155612+0000 mgr.smithi053.fgsmol (mgr.14164) 97 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:22.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:02:22.014772+0000 mon.smithi053 (mon.0) 180 : audit [DBG] from='client.? 172.21.15.146:0/2485848463' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:02:23.433 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:02:23.433 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:02:24.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:24.636+0000 7f1b9b998700 1 -- 172.21.15.146:0/1248695823 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1b940f47c0 msgr2=0x7f1b940f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:24.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:24.636+0000 7f1b9b998700 1 --2- 172.21.15.146:0/1248695823 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1b940f47c0 0x7f1b940f4be0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f1b84009fb0 tx=0x7f1b84009b30).stop 2024-01-21T20:02:24.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:24.636+0000 7f1b9b998700 1 -- 172.21.15.146:0/1248695823 shutdown_connections 2024-01-21T20:02:24.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:24.636+0000 7f1b9b998700 1 --2- 172.21.15.146:0/1248695823 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1b940f47c0 0x7f1b940f4be0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:24.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:24.636+0000 7f1b9b998700 1 -- 172.21.15.146:0/1248695823 >> 172.21.15.146:0/1248695823 conn(0x7f1b940f02e0 msgr2=0x7f1b940f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:24.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:24.637+0000 7f1b9b998700 1 -- 172.21.15.146:0/1248695823 shutdown_connections 2024-01-21T20:02:24.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:24.637+0000 7f1b9b998700 1 -- 172.21.15.146:0/1248695823 wait complete. 2024-01-21T20:02:24.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:24.637+0000 7f1b9b998700 1 Processor -- start 2024-01-21T20:02:24.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:24.637+0000 7f1b9b998700 1 -- start start 2024-01-21T20:02:24.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:24.637+0000 7f1b9b998700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1b940f47c0 0x7f1b94106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:24.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:24.637+0000 7f1b9b998700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b94106bf0 con 0x7f1b940f47c0 2024-01-21T20:02:24.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:24.637+0000 7f1b99734700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1b940f47c0 0x7f1b94106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:24.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:24.638+0000 7f1b99734700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1b940f47c0 0x7f1b94106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:53848/0 (socket says 172.21.15.146:53848) 2024-01-21T20:02:24.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:24.638+0000 7f1b99734700 1 -- 172.21.15.146:0/2421041768 learned_addr learned my addr 172.21.15.146:0/2421041768 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:02:24.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:24.638+0000 7f1b99734700 1 -- 172.21.15.146:0/2421041768 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b84005040 con 0x7f1b940f47c0 2024-01-21T20:02:24.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:24.638+0000 7f1b99734700 1 --2- 172.21.15.146:0/2421041768 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1b940f47c0 0x7f1b94106700 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f1b84013510 tx=0x7f1b8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:24.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:24.638+0000 7f1b89ffb700 1 -- 172.21.15.146:0/2421041768 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b84009320 con 0x7f1b940f47c0 2024-01-21T20:02:24.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:24.638+0000 7f1b89ffb700 1 -- 172.21.15.146:0/2421041768 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1b84013f00 con 0x7f1b940f47c0 2024-01-21T20:02:24.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:24.638+0000 7f1b89ffb700 1 -- 172.21.15.146:0/2421041768 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b84014d90 con 0x7f1b940f47c0 2024-01-21T20:02:24.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:24.638+0000 7f1b9b998700 1 -- 172.21.15.146:0/2421041768 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b94106df0 con 0x7f1b940f47c0 2024-01-21T20:02:24.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:24.638+0000 7f1b9b998700 1 -- 172.21.15.146:0/2421041768 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b94107280 con 0x7f1b940f47c0 2024-01-21T20:02:24.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:24.639+0000 7f1b89ffb700 1 -- 172.21.15.146:0/2421041768 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1b840095f0 con 0x7f1b940f47c0 2024-01-21T20:02:24.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:24.639+0000 7f1b9b998700 1 -- 172.21.15.146:0/2421041768 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b94045ad0 con 0x7f1b940f47c0 2024-01-21T20:02:24.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:24.639+0000 7f1b89ffb700 1 --2- 172.21.15.146:0/2421041768 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f1b8002ffb0 0x7f1b80032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:24.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:24.639+0000 7f1b89ffb700 1 -- 172.21.15.146:0/2421041768 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1b84041050 con 0x7f1b940f47c0 2024-01-21T20:02:24.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:24.639+0000 7f1b98f33700 1 --2- 172.21.15.146:0/2421041768 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f1b8002ffb0 0x7f1b80032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:24.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:24.640+0000 7f1b98f33700 1 --2- 172.21.15.146:0/2421041768 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f1b8002ffb0 0x7f1b80032470 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f1b900096a0 tx=0x7f1b90006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:24.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:24.644+0000 7f1b89ffb700 1 -- 172.21.15.146:0/2421041768 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1b84014420 con 0x7f1b940f47c0 2024-01-21T20:02:24.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:24.848+0000 7f1b9b998700 1 -- 172.21.15.146:0/2421041768 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1b94108130 con 0x7f1b940f47c0 2024-01-21T20:02:24.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:24.849+0000 7f1b89ffb700 1 -- 172.21.15.146:0/2421041768 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1b8401d7f0 con 0x7f1b940f47c0 2024-01-21T20:02:24.840 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:02:24.840 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:02:24.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:24.851+0000 7f1b9b998700 1 -- 172.21.15.146:0/2421041768 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f1b8002ffb0 msgr2=0x7f1b80032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:24.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:24.851+0000 7f1b9b998700 1 --2- 172.21.15.146:0/2421041768 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f1b8002ffb0 0x7f1b80032470 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f1b900096a0 tx=0x7f1b90006b40).stop 2024-01-21T20:02:24.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:24.851+0000 7f1b9b998700 1 -- 172.21.15.146:0/2421041768 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1b940f47c0 msgr2=0x7f1b94106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:24.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:24.851+0000 7f1b9b998700 1 --2- 172.21.15.146:0/2421041768 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1b940f47c0 0x7f1b94106700 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f1b84013510 tx=0x7f1b8400b040).stop 2024-01-21T20:02:24.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:24.851+0000 7f1b9b998700 1 -- 172.21.15.146:0/2421041768 shutdown_connections 2024-01-21T20:02:24.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:24.851+0000 7f1b9b998700 1 --2- 172.21.15.146:0/2421041768 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f1b8002ffb0 0x7f1b80032470 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:24.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:24.851+0000 7f1b9b998700 1 --2- 172.21.15.146:0/2421041768 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1b940f47c0 0x7f1b94106700 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:24.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:24.851+0000 7f1b9b998700 1 -- 172.21.15.146:0/2421041768 >> 172.21.15.146:0/2421041768 conn(0x7f1b940f02e0 msgr2=0x7f1b940f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:24.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:24.851+0000 7f1b9b998700 1 -- 172.21.15.146:0/2421041768 shutdown_connections 2024-01-21T20:02:24.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:24.851+0000 7f1b9b998700 1 -- 172.21.15.146:0/2421041768 wait complete. 2024-01-21T20:02:24.843 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:02:24.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:24 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:02:23.155956+0000 mgr.smithi053.fgsmol (mgr.14164) 98 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:25.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:25 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:02:24.850329+0000 mon.smithi053 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.146:0/2421041768' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:02:26.242 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:02:26.243 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:02:26.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:26 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:02:25.156294+0000 mgr.smithi053.fgsmol (mgr.14164) 99 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:27.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:27.435+0000 7f9ae8307700 1 -- 172.21.15.146:0/748616195 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ae00f5410 msgr2=0x7f9ae00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:27.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:27.435+0000 7f9ae8307700 1 --2- 172.21.15.146:0/748616195 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ae00f5410 0x7f9ae00f5830 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f9adc009fb0 tx=0x7f9adc009b30).stop 2024-01-21T20:02:27.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:27.435+0000 7f9ae8307700 1 -- 172.21.15.146:0/748616195 shutdown_connections 2024-01-21T20:02:27.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:27.435+0000 7f9ae8307700 1 --2- 172.21.15.146:0/748616195 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ae00f5410 0x7f9ae00f5830 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:27.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:27.435+0000 7f9ae8307700 1 -- 172.21.15.146:0/748616195 >> 172.21.15.146:0/748616195 conn(0x7f9ae00f0b70 msgr2=0x7f9ae00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:27.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:27.435+0000 7f9ae8307700 1 -- 172.21.15.146:0/748616195 shutdown_connections 2024-01-21T20:02:27.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:27.435+0000 7f9ae8307700 1 -- 172.21.15.146:0/748616195 wait complete. 2024-01-21T20:02:27.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:27.435+0000 7f9ae8307700 1 Processor -- start 2024-01-21T20:02:27.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:27.436+0000 7f9ae8307700 1 -- start start 2024-01-21T20:02:27.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:27.436+0000 7f9ae8307700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ae00f5410 0x7f9ae0109420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:27.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:27.436+0000 7f9ae8307700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae0060220 con 0x7f9ae00f5410 2024-01-21T20:02:27.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:27.436+0000 7f9ae60a3700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ae00f5410 0x7f9ae0109420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:27.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:27.436+0000 7f9ae60a3700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ae00f5410 0x7f9ae0109420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:35728/0 (socket says 172.21.15.146:35728) 2024-01-21T20:02:27.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:27.436+0000 7f9ae60a3700 1 -- 172.21.15.146:0/908271614 learned_addr learned my addr 172.21.15.146:0/908271614 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:02:27.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:27.436+0000 7f9ae60a3700 1 -- 172.21.15.146:0/908271614 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9adc005040 con 0x7f9ae00f5410 2024-01-21T20:02:27.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:27.436+0000 7f9ae60a3700 1 --2- 172.21.15.146:0/908271614 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ae00f5410 0x7f9ae0109420 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f9adc013510 tx=0x7f9adc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:27.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:27.436+0000 7f9ad6ffd700 1 -- 172.21.15.146:0/908271614 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9adc009320 con 0x7f9ae00f5410 2024-01-21T20:02:27.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:27.436+0000 7f9ae8307700 1 -- 172.21.15.146:0/908271614 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ae0060420 con 0x7f9ae00f5410 2024-01-21T20:02:27.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:27.436+0000 7f9ad6ffd700 1 -- 172.21.15.146:0/908271614 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9adc013f00 con 0x7f9ae00f5410 2024-01-21T20:02:27.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:27.437+0000 7f9ad6ffd700 1 -- 172.21.15.146:0/908271614 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9adc014d90 con 0x7f9ae00f5410 2024-01-21T20:02:27.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:27.437+0000 7f9ae8307700 1 -- 172.21.15.146:0/908271614 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ae0109ad0 con 0x7f9ae00f5410 2024-01-21T20:02:27.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:27.437+0000 7f9ad6ffd700 1 -- 172.21.15.146:0/908271614 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9adc014420 con 0x7f9ae00f5410 2024-01-21T20:02:27.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:27.437+0000 7f9ae8307700 1 -- 172.21.15.146:0/908271614 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ae0059100 con 0x7f9ae00f5410 2024-01-21T20:02:27.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:27.437+0000 7f9ad6ffd700 1 --2- 172.21.15.146:0/908271614 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f9acc02ff60 0x7f9acc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:27.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:27.438+0000 7f9ad6ffd700 1 -- 172.21.15.146:0/908271614 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9adc041080 con 0x7f9ae00f5410 2024-01-21T20:02:27.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:27.438+0000 7f9ae58a2700 1 --2- 172.21.15.146:0/908271614 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f9acc02ff60 0x7f9acc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:27.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:27.438+0000 7f9ae58a2700 1 --2- 172.21.15.146:0/908271614 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f9acc02ff60 0x7f9acc032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f9ad00096a0 tx=0x7f9ad0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:27.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:27.443+0000 7f9ad6ffd700 1 -- 172.21.15.146:0/908271614 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9adc0095f0 con 0x7f9ae00f5410 2024-01-21T20:02:27.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:27.648+0000 7f9ae8307700 1 -- 172.21.15.146:0/908271614 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9ae0045ad0 con 0x7f9ae00f5410 2024-01-21T20:02:27.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:27.648+0000 7f9ad6ffd700 1 -- 172.21.15.146:0/908271614 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9adc01db70 con 0x7f9ae00f5410 2024-01-21T20:02:27.640 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:02:27.640 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:02:27.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:27.650+0000 7f9ae8307700 1 -- 172.21.15.146:0/908271614 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f9acc02ff60 msgr2=0x7f9acc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:27.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:27.650+0000 7f9ae8307700 1 --2- 172.21.15.146:0/908271614 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f9acc02ff60 0x7f9acc032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f9ad00096a0 tx=0x7f9ad0006b40).stop 2024-01-21T20:02:27.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:27.650+0000 7f9ae8307700 1 -- 172.21.15.146:0/908271614 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ae00f5410 msgr2=0x7f9ae0109420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:27.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:27.650+0000 7f9ae8307700 1 --2- 172.21.15.146:0/908271614 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ae00f5410 0x7f9ae0109420 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f9adc013510 tx=0x7f9adc00b040).stop 2024-01-21T20:02:27.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:27.650+0000 7f9ae8307700 1 -- 172.21.15.146:0/908271614 shutdown_connections 2024-01-21T20:02:27.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:27.650+0000 7f9ae8307700 1 --2- 172.21.15.146:0/908271614 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f9acc02ff60 0x7f9acc032420 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:27.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:27.650+0000 7f9ae8307700 1 --2- 172.21.15.146:0/908271614 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ae00f5410 0x7f9ae0109420 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:27.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:27.650+0000 7f9ae8307700 1 -- 172.21.15.146:0/908271614 >> 172.21.15.146:0/908271614 conn(0x7f9ae00f0b70 msgr2=0x7f9ae00f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:27.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:27.650+0000 7f9ae8307700 1 -- 172.21.15.146:0/908271614 shutdown_connections 2024-01-21T20:02:27.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:27.650+0000 7f9ae8307700 1 -- 172.21.15.146:0/908271614 wait complete. 2024-01-21T20:02:27.642 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:02:27.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:27 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:02:27.649868+0000 mon.smithi053 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.146:0/908271614' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:02:28.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:28 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:02:27.156513+0000 mgr.smithi053.fgsmol (mgr.14164) 100 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:29.083 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:02:29.083 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:02:30.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:30.219+0000 7feb8484b700 1 -- 172.21.15.146:0/776190908 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feb800601e0 msgr2=0x7feb800f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:30.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:30.219+0000 7feb8484b700 1 --2- 172.21.15.146:0/776190908 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feb800601e0 0x7feb800f8f20 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7feb70009fb0 tx=0x7feb70009b30).stop 2024-01-21T20:02:30.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:30.219+0000 7feb8484b700 1 -- 172.21.15.146:0/776190908 shutdown_connections 2024-01-21T20:02:30.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:30.219+0000 7feb8484b700 1 --2- 172.21.15.146:0/776190908 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feb800601e0 0x7feb800f8f20 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:30.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:30.219+0000 7feb8484b700 1 -- 172.21.15.146:0/776190908 >> 172.21.15.146:0/776190908 conn(0x7feb800f09a0 msgr2=0x7feb800f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:30.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:30.219+0000 7feb8484b700 1 -- 172.21.15.146:0/776190908 shutdown_connections 2024-01-21T20:02:30.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:30.219+0000 7feb8484b700 1 -- 172.21.15.146:0/776190908 wait complete. 2024-01-21T20:02:30.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:30.219+0000 7feb8484b700 1 Processor -- start 2024-01-21T20:02:30.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:30.219+0000 7feb8484b700 1 -- start start 2024-01-21T20:02:30.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:30.220+0000 7feb8484b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feb800601e0 0x7feb800f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:30.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:30.220+0000 7feb8484b700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb800f4e20 con 0x7feb800601e0 2024-01-21T20:02:30.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:30.220+0000 7feb7ed9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feb800601e0 0x7feb800f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:30.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:30.220+0000 7feb7ed9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feb800601e0 0x7feb800f69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:35746/0 (socket says 172.21.15.146:35746) 2024-01-21T20:02:30.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:30.220+0000 7feb7ed9d700 1 -- 172.21.15.146:0/791541164 learned_addr learned my addr 172.21.15.146:0/791541164 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:02:30.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:30.220+0000 7feb7ed9d700 1 -- 172.21.15.146:0/791541164 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb70005040 con 0x7feb800601e0 2024-01-21T20:02:30.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:30.220+0000 7feb7ed9d700 1 --2- 172.21.15.146:0/791541164 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feb800601e0 0x7feb800f69a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7feb70013510 tx=0x7feb7000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:30.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:30.220+0000 7feb677fe700 1 -- 172.21.15.146:0/791541164 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb70009320 con 0x7feb800601e0 2024-01-21T20:02:30.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:30.220+0000 7feb8484b700 1 -- 172.21.15.146:0/791541164 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb800f4fc0 con 0x7feb800601e0 2024-01-21T20:02:30.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:30.220+0000 7feb677fe700 1 -- 172.21.15.146:0/791541164 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feb70013f00 con 0x7feb800601e0 2024-01-21T20:02:30.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:30.220+0000 7feb677fe700 1 -- 172.21.15.146:0/791541164 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb70014d90 con 0x7feb800601e0 2024-01-21T20:02:30.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:30.220+0000 7feb8484b700 1 -- 172.21.15.146:0/791541164 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb800f5450 con 0x7feb800601e0 2024-01-21T20:02:30.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:30.221+0000 7feb677fe700 1 -- 172.21.15.146:0/791541164 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feb70014420 con 0x7feb800601e0 2024-01-21T20:02:30.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:30.221+0000 7feb8484b700 1 -- 172.21.15.146:0/791541164 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb80059100 con 0x7feb800601e0 2024-01-21T20:02:30.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:30.221+0000 7feb677fe700 1 --2- 172.21.15.146:0/791541164 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7feb6802ff60 0x7feb68032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:30.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:30.221+0000 7feb677fe700 1 -- 172.21.15.146:0/791541164 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feb70041000 con 0x7feb800601e0 2024-01-21T20:02:30.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:30.222+0000 7feb7e59c700 1 --2- 172.21.15.146:0/791541164 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7feb6802ff60 0x7feb68032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:30.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:30.222+0000 7feb7e59c700 1 --2- 172.21.15.146:0/791541164 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7feb6802ff60 0x7feb68032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7feb740096a0 tx=0x7feb74006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:30.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:30.226+0000 7feb677fe700 1 -- 172.21.15.146:0/791541164 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feb700095f0 con 0x7feb800601e0 2024-01-21T20:02:30.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:30.432+0000 7feb8484b700 1 -- 172.21.15.146:0/791541164 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feb800fbcd0 con 0x7feb800601e0 2024-01-21T20:02:30.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:30.433+0000 7feb677fe700 1 -- 172.21.15.146:0/791541164 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7feb7001db70 con 0x7feb800601e0 2024-01-21T20:02:30.424 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:02:30.424 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:02:30.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:30.434+0000 7feb8484b700 1 -- 172.21.15.146:0/791541164 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7feb6802ff60 msgr2=0x7feb68032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:30.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:30.434+0000 7feb8484b700 1 --2- 172.21.15.146:0/791541164 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7feb6802ff60 0x7feb68032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7feb740096a0 tx=0x7feb74006b40).stop 2024-01-21T20:02:30.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:30.434+0000 7feb8484b700 1 -- 172.21.15.146:0/791541164 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feb800601e0 msgr2=0x7feb800f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:30.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:30.434+0000 7feb8484b700 1 --2- 172.21.15.146:0/791541164 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feb800601e0 0x7feb800f69a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7feb70013510 tx=0x7feb7000b040).stop 2024-01-21T20:02:30.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:30.434+0000 7feb8484b700 1 -- 172.21.15.146:0/791541164 shutdown_connections 2024-01-21T20:02:30.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:30.434+0000 7feb8484b700 1 --2- 172.21.15.146:0/791541164 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7feb6802ff60 0x7feb68032420 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:30.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:30.434+0000 7feb8484b700 1 --2- 172.21.15.146:0/791541164 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feb800601e0 0x7feb800f69a0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:30.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:30.434+0000 7feb8484b700 1 -- 172.21.15.146:0/791541164 >> 172.21.15.146:0/791541164 conn(0x7feb800f09a0 msgr2=0x7feb800f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:30.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:30.434+0000 7feb8484b700 1 -- 172.21.15.146:0/791541164 shutdown_connections 2024-01-21T20:02:30.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:30.434+0000 7feb8484b700 1 -- 172.21.15.146:0/791541164 wait complete. 2024-01-21T20:02:30.426 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:02:30.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:30 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:02:29.156842+0000 mgr.smithi053.fgsmol (mgr.14164) 101 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:30.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:30 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:02:30.432918+0000 mon.smithi053 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.146:0/791541164' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:02:31.858 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:02:31.858 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:02:32.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:32 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:02:31.157242+0000 mgr.smithi053.fgsmol (mgr.14164) 102 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:33.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:33.132+0000 7fe27af4c700 1 -- 172.21.15.146:0/4146276681 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2740f78e0 msgr2=0x7fe2740f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:33.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:33.132+0000 7fe27af4c700 1 --2- 172.21.15.146:0/4146276681 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2740f78e0 0x7fe2740f7d00 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7fe270009fb0 tx=0x7fe270009b30).stop 2024-01-21T20:02:33.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:33.132+0000 7fe27af4c700 1 -- 172.21.15.146:0/4146276681 shutdown_connections 2024-01-21T20:02:33.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:33.132+0000 7fe27af4c700 1 --2- 172.21.15.146:0/4146276681 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2740f78e0 0x7fe2740f7d00 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:33.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:33.132+0000 7fe27af4c700 1 -- 172.21.15.146:0/4146276681 >> 172.21.15.146:0/4146276681 conn(0x7fe2740f0120 msgr2=0x7fe2740f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:33.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:33.132+0000 7fe27af4c700 1 -- 172.21.15.146:0/4146276681 shutdown_connections 2024-01-21T20:02:33.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:33.132+0000 7fe27af4c700 1 -- 172.21.15.146:0/4146276681 wait complete. 2024-01-21T20:02:33.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:33.133+0000 7fe27af4c700 1 Processor -- start 2024-01-21T20:02:33.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:33.133+0000 7fe27af4c700 1 -- start start 2024-01-21T20:02:33.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:33.133+0000 7fe27af4c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2740f78e0 0x7fe2741064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:33.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:33.133+0000 7fe27af4c700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2741069d0 con 0x7fe2740f78e0 2024-01-21T20:02:33.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:33.133+0000 7fe279f4a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2740f78e0 0x7fe2741064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:33.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:33.133+0000 7fe279f4a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2740f78e0 0x7fe2741064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:35772/0 (socket says 172.21.15.146:35772) 2024-01-21T20:02:33.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:33.133+0000 7fe279f4a700 1 -- 172.21.15.146:0/884581655 learned_addr learned my addr 172.21.15.146:0/884581655 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:02:33.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:33.133+0000 7fe279f4a700 1 -- 172.21.15.146:0/884581655 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe270005040 con 0x7fe2740f78e0 2024-01-21T20:02:33.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:33.133+0000 7fe279f4a700 1 --2- 172.21.15.146:0/884581655 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2740f78e0 0x7fe2741064e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fe270009830 tx=0x7fe27000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:33.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:33.134+0000 7fe26a7fc700 1 -- 172.21.15.146:0/884581655 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe270009320 con 0x7fe2740f78e0 2024-01-21T20:02:33.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:33.134+0000 7fe26a7fc700 1 -- 172.21.15.146:0/884581655 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe2700136c0 con 0x7fe2740f78e0 2024-01-21T20:02:33.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:33.134+0000 7fe27af4c700 1 -- 172.21.15.146:0/884581655 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe274106bd0 con 0x7fe2740f78e0 2024-01-21T20:02:33.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:33.134+0000 7fe26a7fc700 1 -- 172.21.15.146:0/884581655 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe27001ee90 con 0x7fe2740f78e0 2024-01-21T20:02:33.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:33.134+0000 7fe27af4c700 1 -- 172.21.15.146:0/884581655 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe274107060 con 0x7fe2740f78e0 2024-01-21T20:02:33.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:33.134+0000 7fe26a7fc700 1 -- 172.21.15.146:0/884581655 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe270014440 con 0x7fe2740f78e0 2024-01-21T20:02:33.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:33.135+0000 7fe26a7fc700 1 --2- 172.21.15.146:0/884581655 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fe26002ff10 0x7fe2600323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:33.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:33.135+0000 7fe26a7fc700 1 -- 172.21.15.146:0/884581655 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe270041320 con 0x7fe2740f78e0 2024-01-21T20:02:33.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:33.135+0000 7fe27af4c700 1 -- 172.21.15.146:0/884581655 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe274045ad0 con 0x7fe2740f78e0 2024-01-21T20:02:33.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:33.135+0000 7fe279749700 1 --2- 172.21.15.146:0/884581655 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fe26002ff10 0x7fe2600323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:33.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:33.135+0000 7fe279749700 1 --2- 172.21.15.146:0/884581655 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fe26002ff10 0x7fe2600323d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fe2640096a0 tx=0x7fe264006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:33.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:33.140+0000 7fe26a7fc700 1 -- 172.21.15.146:0/884581655 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe2700273f0 con 0x7fe2740f78e0 2024-01-21T20:02:33.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:33.343+0000 7fe27af4c700 1 -- 172.21.15.146:0/884581655 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe2740fbdd0 con 0x7fe2740f78e0 2024-01-21T20:02:33.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:33.344+0000 7fe26a7fc700 1 -- 172.21.15.146:0/884581655 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe27001c320 con 0x7fe2740f78e0 2024-01-21T20:02:33.334 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:02:33.334 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:02:33.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:33.345+0000 7fe27af4c700 1 -- 172.21.15.146:0/884581655 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fe26002ff10 msgr2=0x7fe2600323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:33.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:33.345+0000 7fe27af4c700 1 --2- 172.21.15.146:0/884581655 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fe26002ff10 0x7fe2600323d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fe2640096a0 tx=0x7fe264006b40).stop 2024-01-21T20:02:33.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:33.345+0000 7fe27af4c700 1 -- 172.21.15.146:0/884581655 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2740f78e0 msgr2=0x7fe2741064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:33.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:33.345+0000 7fe27af4c700 1 --2- 172.21.15.146:0/884581655 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2740f78e0 0x7fe2741064e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fe270009830 tx=0x7fe27000b040).stop 2024-01-21T20:02:33.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:33.345+0000 7fe27af4c700 1 -- 172.21.15.146:0/884581655 shutdown_connections 2024-01-21T20:02:33.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:33.345+0000 7fe27af4c700 1 --2- 172.21.15.146:0/884581655 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fe26002ff10 0x7fe2600323d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:33.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:33.345+0000 7fe27af4c700 1 --2- 172.21.15.146:0/884581655 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2740f78e0 0x7fe2741064e0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:33.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:33.345+0000 7fe27af4c700 1 -- 172.21.15.146:0/884581655 >> 172.21.15.146:0/884581655 conn(0x7fe2740f0120 msgr2=0x7fe2740f0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:33.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:33.345+0000 7fe27af4c700 1 -- 172.21.15.146:0/884581655 shutdown_connections 2024-01-21T20:02:33.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:33.345+0000 7fe27af4c700 1 -- 172.21.15.146:0/884581655 wait complete. 2024-01-21T20:02:33.337 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:02:33.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:33 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:02:33.343674+0000 mon.smithi053 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.146:0/884581655' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:02:34.771 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:02:34.771 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:02:34.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:34 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:02:33.157488+0000 mgr.smithi053.fgsmol (mgr.14164) 103 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:35.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:35.978+0000 7fae7359e700 1 -- 172.21.15.146:0/2442625471 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae740f67d0 msgr2=0x7fae740f6bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:35.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:35.978+0000 7fae7359e700 1 --2- 172.21.15.146:0/2442625471 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae740f67d0 0x7fae740f6bf0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7fae64009fb0 tx=0x7fae64009b30).stop 2024-01-21T20:02:35.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:35.978+0000 7fae7359e700 1 -- 172.21.15.146:0/2442625471 shutdown_connections 2024-01-21T20:02:35.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:35.978+0000 7fae7359e700 1 --2- 172.21.15.146:0/2442625471 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae740f67d0 0x7fae740f6bf0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:35.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:35.978+0000 7fae7359e700 1 -- 172.21.15.146:0/2442625471 >> 172.21.15.146:0/2442625471 conn(0x7fae740effc0 msgr2=0x7fae740f23e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:35.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:35.978+0000 7fae7359e700 1 -- 172.21.15.146:0/2442625471 shutdown_connections 2024-01-21T20:02:35.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:35.978+0000 7fae7359e700 1 -- 172.21.15.146:0/2442625471 wait complete. 2024-01-21T20:02:35.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:35.978+0000 7fae7359e700 1 Processor -- start 2024-01-21T20:02:35.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:35.978+0000 7fae7359e700 1 -- start start 2024-01-21T20:02:35.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:35.979+0000 7fae7359e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae740f67d0 0x7fae74109170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:35.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:35.979+0000 7fae7359e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae740f5010 con 0x7fae740f67d0 2024-01-21T20:02:35.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:35.979+0000 7fae7259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae740f67d0 0x7fae74109170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:35.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:35.979+0000 7fae7259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae740f67d0 0x7fae74109170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:35788/0 (socket says 172.21.15.146:35788) 2024-01-21T20:02:35.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:35.979+0000 7fae7259c700 1 -- 172.21.15.146:0/1910393486 learned_addr learned my addr 172.21.15.146:0/1910393486 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:02:35.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:35.979+0000 7fae7259c700 1 -- 172.21.15.146:0/1910393486 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae64005040 con 0x7fae740f67d0 2024-01-21T20:02:35.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:35.979+0000 7fae7259c700 1 --2- 172.21.15.146:0/1910393486 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae740f67d0 0x7fae74109170 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fae640089f0 tx=0x7fae6400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:35.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:35.980+0000 7fae62ffd700 1 -- 172.21.15.146:0/1910393486 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae64009320 con 0x7fae740f67d0 2024-01-21T20:02:35.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:35.980+0000 7fae7359e700 1 -- 172.21.15.146:0/1910393486 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae740f5210 con 0x7fae740f67d0 2024-01-21T20:02:35.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:35.980+0000 7fae62ffd700 1 -- 172.21.15.146:0/1910393486 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fae640136c0 con 0x7fae740f67d0 2024-01-21T20:02:35.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:35.980+0000 7fae62ffd700 1 -- 172.21.15.146:0/1910393486 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae6401d380 con 0x7fae740f67d0 2024-01-21T20:02:35.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:35.980+0000 7fae7359e700 1 -- 172.21.15.146:0/1910393486 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae74109820 con 0x7fae740f67d0 2024-01-21T20:02:35.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:35.980+0000 7fae62ffd700 1 -- 172.21.15.146:0/1910393486 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fae64014420 con 0x7fae740f67d0 2024-01-21T20:02:35.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:35.980+0000 7fae7359e700 1 -- 172.21.15.146:0/1910393486 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae74059100 con 0x7fae740f67d0 2024-01-21T20:02:35.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:35.981+0000 7fae62ffd700 1 --2- 172.21.15.146:0/1910393486 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fae5c02ffb0 0x7fae5c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:35.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:35.981+0000 7fae62ffd700 1 -- 172.21.15.146:0/1910393486 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fae640410e0 con 0x7fae740f67d0 2024-01-21T20:02:35.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:35.981+0000 7fae71d9b700 1 --2- 172.21.15.146:0/1910393486 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fae5c02ffb0 0x7fae5c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:35.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:35.981+0000 7fae71d9b700 1 --2- 172.21.15.146:0/1910393486 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fae5c02ffb0 0x7fae5c032470 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fae680096a0 tx=0x7fae68006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:35.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:35.986+0000 7fae62ffd700 1 -- 172.21.15.146:0/1910393486 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fae64012330 con 0x7fae740f67d0 2024-01-21T20:02:36.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:36.195+0000 7fae7359e700 1 -- 172.21.15.146:0/1910393486 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fae74045ad0 con 0x7fae740f67d0 2024-01-21T20:02:36.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:36.196+0000 7fae62ffd700 1 -- 172.21.15.146:0/1910393486 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fae6401cb10 con 0x7fae740f67d0 2024-01-21T20:02:36.186 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:02:36.186 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:02:36.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:36.197+0000 7fae7359e700 1 -- 172.21.15.146:0/1910393486 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fae5c02ffb0 msgr2=0x7fae5c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:36.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:36.197+0000 7fae7359e700 1 --2- 172.21.15.146:0/1910393486 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fae5c02ffb0 0x7fae5c032470 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fae680096a0 tx=0x7fae68006b40).stop 2024-01-21T20:02:36.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:36.197+0000 7fae7359e700 1 -- 172.21.15.146:0/1910393486 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae740f67d0 msgr2=0x7fae74109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:36.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:36.197+0000 7fae7359e700 1 --2- 172.21.15.146:0/1910393486 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae740f67d0 0x7fae74109170 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fae640089f0 tx=0x7fae6400b040).stop 2024-01-21T20:02:36.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:36.197+0000 7fae7359e700 1 -- 172.21.15.146:0/1910393486 shutdown_connections 2024-01-21T20:02:36.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:36.197+0000 7fae7359e700 1 --2- 172.21.15.146:0/1910393486 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fae5c02ffb0 0x7fae5c032470 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:36.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:36.197+0000 7fae7359e700 1 --2- 172.21.15.146:0/1910393486 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae740f67d0 0x7fae74109170 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:36.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:36.197+0000 7fae7359e700 1 -- 172.21.15.146:0/1910393486 >> 172.21.15.146:0/1910393486 conn(0x7fae740effc0 msgr2=0x7fae740f0a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:36.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:36.197+0000 7fae7359e700 1 -- 172.21.15.146:0/1910393486 shutdown_connections 2024-01-21T20:02:36.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:36.197+0000 7fae7359e700 1 -- 172.21.15.146:0/1910393486 wait complete. 2024-01-21T20:02:36.189 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:02:36.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:36 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:02:35.157759+0000 mgr.smithi053.fgsmol (mgr.14164) 104 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:36.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:36 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:02:36.196037+0000 mon.smithi053 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.146:0/1910393486' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:02:37.603 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:02:37.603 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:02:38.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:38.814+0000 7f0d0759e700 1 -- 172.21.15.146:0/280124672 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d080f51a0 msgr2=0x7f0d080f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:38.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:38.814+0000 7f0d0759e700 1 --2- 172.21.15.146:0/280124672 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d080f51a0 0x7f0d080f55c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f0cfc009fb0 tx=0x7f0cfc009b30).stop 2024-01-21T20:02:38.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:38.814+0000 7f0d0759e700 1 -- 172.21.15.146:0/280124672 shutdown_connections 2024-01-21T20:02:38.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:38.814+0000 7f0d0759e700 1 --2- 172.21.15.146:0/280124672 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d080f51a0 0x7f0d080f55c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:38.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:38.814+0000 7f0d0759e700 1 -- 172.21.15.146:0/280124672 >> 172.21.15.146:0/280124672 conn(0x7f0d080f09a0 msgr2=0x7f0d080f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:38.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:38.814+0000 7f0d0759e700 1 -- 172.21.15.146:0/280124672 shutdown_connections 2024-01-21T20:02:38.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:38.814+0000 7f0d0759e700 1 -- 172.21.15.146:0/280124672 wait complete. 2024-01-21T20:02:38.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:38.814+0000 7f0d0759e700 1 Processor -- start 2024-01-21T20:02:38.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:38.814+0000 7f0d0759e700 1 -- start start 2024-01-21T20:02:38.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:38.814+0000 7f0d0759e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d080f51a0 0x7f0d081091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:38.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:38.814+0000 7f0d0759e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d08060240 con 0x7f0d080f51a0 2024-01-21T20:02:38.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:38.815+0000 7f0d0659c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d080f51a0 0x7f0d081091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:38.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:38.815+0000 7f0d0659c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d080f51a0 0x7f0d081091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:43000/0 (socket says 172.21.15.146:43000) 2024-01-21T20:02:38.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:38.815+0000 7f0d0659c700 1 -- 172.21.15.146:0/2664791207 learned_addr learned my addr 172.21.15.146:0/2664791207 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:02:38.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:38.815+0000 7f0d0659c700 1 -- 172.21.15.146:0/2664791207 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cfc005040 con 0x7f0d080f51a0 2024-01-21T20:02:38.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:38.815+0000 7f0d0659c700 1 --2- 172.21.15.146:0/2664791207 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d080f51a0 0x7f0d081091b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f0cfc013510 tx=0x7f0cfc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:38.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:38.815+0000 7f0cf6ffd700 1 -- 172.21.15.146:0/2664791207 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cfc009320 con 0x7f0d080f51a0 2024-01-21T20:02:38.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:38.815+0000 7f0d0759e700 1 -- 172.21.15.146:0/2664791207 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d08060440 con 0x7f0d080f51a0 2024-01-21T20:02:38.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:38.815+0000 7f0cf6ffd700 1 -- 172.21.15.146:0/2664791207 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0cfc013f00 con 0x7f0d080f51a0 2024-01-21T20:02:38.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:38.815+0000 7f0cf6ffd700 1 -- 172.21.15.146:0/2664791207 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cfc014dc0 con 0x7f0d080f51a0 2024-01-21T20:02:38.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:38.815+0000 7f0d0759e700 1 -- 172.21.15.146:0/2664791207 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d08109860 con 0x7f0d080f51a0 2024-01-21T20:02:38.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:38.816+0000 7f0cf6ffd700 1 -- 172.21.15.146:0/2664791207 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0cfc01d3d0 con 0x7f0d080f51a0 2024-01-21T20:02:38.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:38.816+0000 7f0cf6ffd700 1 --2- 172.21.15.146:0/2664791207 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0cf002ff60 0x7f0cf0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:38.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:38.816+0000 7f0cf6ffd700 1 -- 172.21.15.146:0/2664791207 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0cfc041320 con 0x7f0d080f51a0 2024-01-21T20:02:38.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:38.816+0000 7f0d0759e700 1 -- 172.21.15.146:0/2664791207 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d08045ad0 con 0x7f0d080f51a0 2024-01-21T20:02:38.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:38.816+0000 7f0d05d9b700 1 --2- 172.21.15.146:0/2664791207 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0cf002ff60 0x7f0cf0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:38.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:38.817+0000 7f0d05d9b700 1 --2- 172.21.15.146:0/2664791207 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0cf002ff60 0x7f0cf0032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f0cf80096a0 tx=0x7f0cf8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:38.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:38.821+0000 7f0cf6ffd700 1 -- 172.21.15.146:0/2664791207 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0cfc012900 con 0x7f0d080f51a0 2024-01-21T20:02:38.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:38 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:02:37.158098+0000 mgr.smithi053.fgsmol (mgr.14164) 105 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:39.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:39.024+0000 7f0d0759e700 1 -- 172.21.15.146:0/2664791207 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0d08109fe0 con 0x7f0d080f51a0 2024-01-21T20:02:39.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:39.025+0000 7f0cf6ffd700 1 -- 172.21.15.146:0/2664791207 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0cfc02b030 con 0x7f0d080f51a0 2024-01-21T20:02:39.015 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:02:39.015 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:02:39.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:39.027+0000 7f0d0759e700 1 -- 172.21.15.146:0/2664791207 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0cf002ff60 msgr2=0x7f0cf0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:39.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:39.027+0000 7f0d0759e700 1 --2- 172.21.15.146:0/2664791207 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0cf002ff60 0x7f0cf0032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f0cf80096a0 tx=0x7f0cf8006b40).stop 2024-01-21T20:02:39.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:39.027+0000 7f0d0759e700 1 -- 172.21.15.146:0/2664791207 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d080f51a0 msgr2=0x7f0d081091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:39.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:39.027+0000 7f0d0759e700 1 --2- 172.21.15.146:0/2664791207 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d080f51a0 0x7f0d081091b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f0cfc013510 tx=0x7f0cfc00b040).stop 2024-01-21T20:02:39.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:39.027+0000 7f0d0759e700 1 -- 172.21.15.146:0/2664791207 shutdown_connections 2024-01-21T20:02:39.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:39.027+0000 7f0d0759e700 1 --2- 172.21.15.146:0/2664791207 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0cf002ff60 0x7f0cf0032420 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:39.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:39.027+0000 7f0d0759e700 1 --2- 172.21.15.146:0/2664791207 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d080f51a0 0x7f0d081091b0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:39.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:39.027+0000 7f0d0759e700 1 -- 172.21.15.146:0/2664791207 >> 172.21.15.146:0/2664791207 conn(0x7f0d080f09a0 msgr2=0x7f0d080f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:39.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:39.027+0000 7f0d0759e700 1 -- 172.21.15.146:0/2664791207 shutdown_connections 2024-01-21T20:02:39.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:39.027+0000 7f0d0759e700 1 -- 172.21.15.146:0/2664791207 wait complete. 2024-01-21T20:02:39.018 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:02:39.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:39 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:02:39.025296+0000 mon.smithi053 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.146:0/2664791207' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:02:40.360 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:02:40.360 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:02:40.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:40 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:02:39.158470+0000 mgr.smithi053.fgsmol (mgr.14164) 106 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:41.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:41.631+0000 7f0f3b533700 1 -- 172.21.15.146:0/3914198034 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f340601e0 msgr2=0x7f0f340f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:41.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:41.631+0000 7f0f3b533700 1 --2- 172.21.15.146:0/3914198034 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f340601e0 0x7f0f340f9190 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f0f24009fb0 tx=0x7f0f24009b30).stop 2024-01-21T20:02:41.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:41.631+0000 7f0f3b533700 1 -- 172.21.15.146:0/3914198034 shutdown_connections 2024-01-21T20:02:41.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:41.631+0000 7f0f3b533700 1 --2- 172.21.15.146:0/3914198034 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f340601e0 0x7f0f340f9190 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:41.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:41.631+0000 7f0f3b533700 1 -- 172.21.15.146:0/3914198034 >> 172.21.15.146:0/3914198034 conn(0x7f0f340f0b70 msgr2=0x7f0f340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:41.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:41.631+0000 7f0f3b533700 1 -- 172.21.15.146:0/3914198034 shutdown_connections 2024-01-21T20:02:41.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:41.631+0000 7f0f3b533700 1 -- 172.21.15.146:0/3914198034 wait complete. 2024-01-21T20:02:41.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:41.632+0000 7f0f3b533700 1 Processor -- start 2024-01-21T20:02:41.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:41.632+0000 7f0f3b533700 1 -- start start 2024-01-21T20:02:41.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:41.632+0000 7f0f3b533700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f340601e0 0x7f0f341066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:41.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:41.632+0000 7f0f3b533700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f34106bd0 con 0x7f0f340601e0 2024-01-21T20:02:41.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:41.632+0000 7f0f392cf700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f340601e0 0x7f0f341066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:41.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:41.632+0000 7f0f392cf700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f340601e0 0x7f0f341066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:43018/0 (socket says 172.21.15.146:43018) 2024-01-21T20:02:41.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:41.632+0000 7f0f392cf700 1 -- 172.21.15.146:0/1225983492 learned_addr learned my addr 172.21.15.146:0/1225983492 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:02:41.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:41.632+0000 7f0f392cf700 1 -- 172.21.15.146:0/1225983492 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f24005040 con 0x7f0f340601e0 2024-01-21T20:02:41.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:41.632+0000 7f0f392cf700 1 --2- 172.21.15.146:0/1225983492 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f340601e0 0x7f0f341066e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f0f24004dd0 tx=0x7f0f24009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:41.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:41.633+0000 7f0f29ffb700 1 -- 172.21.15.146:0/1225983492 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f24018070 con 0x7f0f340601e0 2024-01-21T20:02:41.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:41.633+0000 7f0f3b533700 1 -- 172.21.15.146:0/1225983492 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f34106dd0 con 0x7f0f340601e0 2024-01-21T20:02:41.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:41.633+0000 7f0f29ffb700 1 -- 172.21.15.146:0/1225983492 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0f24013410 con 0x7f0f340601e0 2024-01-21T20:02:41.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:41.633+0000 7f0f29ffb700 1 -- 172.21.15.146:0/1225983492 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f2401c3c0 con 0x7f0f340601e0 2024-01-21T20:02:41.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:41.633+0000 7f0f3b533700 1 -- 172.21.15.146:0/1225983492 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f34107260 con 0x7f0f340601e0 2024-01-21T20:02:41.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:41.634+0000 7f0f29ffb700 1 -- 172.21.15.146:0/1225983492 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0f2401c630 con 0x7f0f340601e0 2024-01-21T20:02:41.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:41.634+0000 7f0f29ffb700 1 --2- 172.21.15.146:0/1225983492 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0f2002ff60 0x7f0f20032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:41.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:41.634+0000 7f0f29ffb700 1 -- 172.21.15.146:0/1225983492 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0f24041380 con 0x7f0f340601e0 2024-01-21T20:02:41.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:41.634+0000 7f0f3b533700 1 -- 172.21.15.146:0/1225983492 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f34045ad0 con 0x7f0f340601e0 2024-01-21T20:02:41.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:41.634+0000 7f0f38ace700 1 --2- 172.21.15.146:0/1225983492 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0f2002ff60 0x7f0f20032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:41.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:41.634+0000 7f0f38ace700 1 --2- 172.21.15.146:0/1225983492 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0f2002ff60 0x7f0f20032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f0f300096a0 tx=0x7f0f30006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:41.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:41.639+0000 7f0f29ffb700 1 -- 172.21.15.146:0/1225983492 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0f2401ca30 con 0x7f0f340601e0 2024-01-21T20:02:41.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:41.843+0000 7f0f3b533700 1 -- 172.21.15.146:0/1225983492 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0f341080b0 con 0x7f0f340601e0 2024-01-21T20:02:41.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:41.843+0000 7f0f29ffb700 1 -- 172.21.15.146:0/1225983492 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0f2401b9c0 con 0x7f0f340601e0 2024-01-21T20:02:41.834 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:02:41.835 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:02:41.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:41.845+0000 7f0f3b533700 1 -- 172.21.15.146:0/1225983492 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0f2002ff60 msgr2=0x7f0f20032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:41.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:41.845+0000 7f0f3b533700 1 --2- 172.21.15.146:0/1225983492 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0f2002ff60 0x7f0f20032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f0f300096a0 tx=0x7f0f30006b40).stop 2024-01-21T20:02:41.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:41.845+0000 7f0f3b533700 1 -- 172.21.15.146:0/1225983492 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f340601e0 msgr2=0x7f0f341066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:41.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:41.845+0000 7f0f3b533700 1 --2- 172.21.15.146:0/1225983492 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f340601e0 0x7f0f341066e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f0f24004dd0 tx=0x7f0f24009b30).stop 2024-01-21T20:02:41.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:41.845+0000 7f0f3b533700 1 -- 172.21.15.146:0/1225983492 shutdown_connections 2024-01-21T20:02:41.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:41.845+0000 7f0f3b533700 1 --2- 172.21.15.146:0/1225983492 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0f2002ff60 0x7f0f20032420 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:41.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:41.845+0000 7f0f3b533700 1 --2- 172.21.15.146:0/1225983492 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f340601e0 0x7f0f341066e0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:41.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:41.845+0000 7f0f3b533700 1 -- 172.21.15.146:0/1225983492 >> 172.21.15.146:0/1225983492 conn(0x7f0f340f0b70 msgr2=0x7f0f340f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:41.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:41.845+0000 7f0f3b533700 1 -- 172.21.15.146:0/1225983492 shutdown_connections 2024-01-21T20:02:41.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:41.845+0000 7f0f3b533700 1 -- 172.21.15.146:0/1225983492 wait complete. 2024-01-21T20:02:41.837 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:02:42.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:42 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:02:41.158878+0000 mgr.smithi053.fgsmol (mgr.14164) 107 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:42.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:42 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:02:41.844588+0000 mon.smithi053 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.146:0/1225983492' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:02:43.228 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:02:43.228 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:02:44.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:44.563+0000 7fa3b2a85700 1 -- 172.21.15.146:0/1992884998 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3ac060220 msgr2=0x7fa3ac0f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:44.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:44.563+0000 7fa3b2a85700 1 --2- 172.21.15.146:0/1992884998 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3ac060220 0x7fa3ac0f9150 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7fa3a0009fb0 tx=0x7fa3a0009b30).stop 2024-01-21T20:02:44.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:44.563+0000 7fa3b2a85700 1 -- 172.21.15.146:0/1992884998 shutdown_connections 2024-01-21T20:02:44.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:44.563+0000 7fa3b2a85700 1 --2- 172.21.15.146:0/1992884998 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3ac060220 0x7fa3ac0f9150 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:44.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:44.563+0000 7fa3b2a85700 1 -- 172.21.15.146:0/1992884998 >> 172.21.15.146:0/1992884998 conn(0x7fa3ac0f0b70 msgr2=0x7fa3ac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:44.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:44.563+0000 7fa3b2a85700 1 -- 172.21.15.146:0/1992884998 shutdown_connections 2024-01-21T20:02:44.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:44.563+0000 7fa3b2a85700 1 -- 172.21.15.146:0/1992884998 wait complete. 2024-01-21T20:02:44.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:44.563+0000 7fa3b2a85700 1 Processor -- start 2024-01-21T20:02:44.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:44.564+0000 7fa3b2a85700 1 -- start start 2024-01-21T20:02:44.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:44.564+0000 7fa3b2a85700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3ac060220 0x7fa3ac1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:44.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:44.564+0000 7fa3b2a85700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3ac106bd0 con 0x7fa3ac060220 2024-01-21T20:02:44.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:44.564+0000 7fa3b0821700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3ac060220 0x7fa3ac1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:44.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:44.564+0000 7fa3b0821700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3ac060220 0x7fa3ac1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:43034/0 (socket says 172.21.15.146:43034) 2024-01-21T20:02:44.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:44.564+0000 7fa3b0821700 1 -- 172.21.15.146:0/1550914622 learned_addr learned my addr 172.21.15.146:0/1550914622 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:02:44.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:44.565+0000 7fa3b0821700 1 -- 172.21.15.146:0/1550914622 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3a0005040 con 0x7fa3ac060220 2024-01-21T20:02:44.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:44.565+0000 7fa3b0821700 1 --2- 172.21.15.146:0/1550914622 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3ac060220 0x7fa3ac1066e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7fa3a0013510 tx=0x7fa3a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:44.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:44.565+0000 7fa3a97fa700 1 -- 172.21.15.146:0/1550914622 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3a0009320 con 0x7fa3ac060220 2024-01-21T20:02:44.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:44.565+0000 7fa3b2a85700 1 -- 172.21.15.146:0/1550914622 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3ac106dd0 con 0x7fa3ac060220 2024-01-21T20:02:44.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:44.565+0000 7fa3a97fa700 1 -- 172.21.15.146:0/1550914622 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa3a0013f00 con 0x7fa3ac060220 2024-01-21T20:02:44.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:44.565+0000 7fa3a97fa700 1 -- 172.21.15.146:0/1550914622 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3a0014dc0 con 0x7fa3ac060220 2024-01-21T20:02:44.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:44.565+0000 7fa3b2a85700 1 -- 172.21.15.146:0/1550914622 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3ac107260 con 0x7fa3ac060220 2024-01-21T20:02:44.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:44.566+0000 7fa3a97fa700 1 -- 172.21.15.146:0/1550914622 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa3a00095f0 con 0x7fa3ac060220 2024-01-21T20:02:44.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:44.566+0000 7fa3b2a85700 1 -- 172.21.15.146:0/1550914622 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3ac045ad0 con 0x7fa3ac060220 2024-01-21T20:02:44.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:44.566+0000 7fa3a97fa700 1 --2- 172.21.15.146:0/1550914622 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa39c02ff60 0x7fa39c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:44.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:44.566+0000 7fa3a97fa700 1 -- 172.21.15.146:0/1550914622 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa3a00410c0 con 0x7fa3ac060220 2024-01-21T20:02:44.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:44.566+0000 7fa3abfff700 1 --2- 172.21.15.146:0/1550914622 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa39c02ff60 0x7fa39c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:44.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:44.567+0000 7fa3abfff700 1 --2- 172.21.15.146:0/1550914622 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa39c02ff60 0x7fa39c032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fa3980096a0 tx=0x7fa398006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:44.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:44.571+0000 7fa3a97fa700 1 -- 172.21.15.146:0/1550914622 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa3a00268c0 con 0x7fa3ac060220 2024-01-21T20:02:44.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:44.777+0000 7fa3b2a85700 1 -- 172.21.15.146:0/1550914622 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa3ac108110 con 0x7fa3ac060220 2024-01-21T20:02:44.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:44.778+0000 7fa3a97fa700 1 -- 172.21.15.146:0/1550914622 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa3a0010230 con 0x7fa3ac060220 2024-01-21T20:02:44.769 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:02:44.769 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:02:44.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:44.779+0000 7fa3b2a85700 1 -- 172.21.15.146:0/1550914622 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa39c02ff60 msgr2=0x7fa39c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:44.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:44.779+0000 7fa3b2a85700 1 --2- 172.21.15.146:0/1550914622 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa39c02ff60 0x7fa39c032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fa3980096a0 tx=0x7fa398006b40).stop 2024-01-21T20:02:44.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:44.779+0000 7fa3b2a85700 1 -- 172.21.15.146:0/1550914622 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3ac060220 msgr2=0x7fa3ac1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:44.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:44.779+0000 7fa3b2a85700 1 --2- 172.21.15.146:0/1550914622 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3ac060220 0x7fa3ac1066e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7fa3a0013510 tx=0x7fa3a000b040).stop 2024-01-21T20:02:44.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:44.779+0000 7fa3b2a85700 1 -- 172.21.15.146:0/1550914622 shutdown_connections 2024-01-21T20:02:44.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:44.779+0000 7fa3b2a85700 1 --2- 172.21.15.146:0/1550914622 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa39c02ff60 0x7fa39c032420 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:44.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:44.779+0000 7fa3b2a85700 1 --2- 172.21.15.146:0/1550914622 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3ac060220 0x7fa3ac1066e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:44.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:44.779+0000 7fa3b2a85700 1 -- 172.21.15.146:0/1550914622 >> 172.21.15.146:0/1550914622 conn(0x7fa3ac0f0b70 msgr2=0x7fa3ac0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:44.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:44.779+0000 7fa3b2a85700 1 -- 172.21.15.146:0/1550914622 shutdown_connections 2024-01-21T20:02:44.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:44.779+0000 7fa3b2a85700 1 -- 172.21.15.146:0/1550914622 wait complete. 2024-01-21T20:02:44.772 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:02:44.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:44 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:02:43.159272+0000 mgr.smithi053.fgsmol (mgr.14164) 108 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:45.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:45 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:02:44.779099+0000 mon.smithi053 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.146:0/1550914622' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:02:46.117 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:02:46.118 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:02:46.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:46 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:02:45.159614+0000 mgr.smithi053.fgsmol (mgr.14164) 109 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:47.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:47.421+0000 7f3b95bc2700 1 -- 172.21.15.146:0/44559673 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b900f6510 msgr2=0x7f3b900f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:47.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:47.421+0000 7f3b95bc2700 1 --2- 172.21.15.146:0/44559673 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b900f6510 0x7f3b900f6930 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f3b78009fb0 tx=0x7f3b78009b30).stop 2024-01-21T20:02:47.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:47.421+0000 7f3b95bc2700 1 -- 172.21.15.146:0/44559673 shutdown_connections 2024-01-21T20:02:47.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:47.421+0000 7f3b95bc2700 1 --2- 172.21.15.146:0/44559673 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b900f6510 0x7f3b900f6930 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:47.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:47.421+0000 7f3b95bc2700 1 -- 172.21.15.146:0/44559673 >> 172.21.15.146:0/44559673 conn(0x7f3b900f1c70 msgr2=0x7f3b900f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:47.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:47.421+0000 7f3b95bc2700 1 -- 172.21.15.146:0/44559673 shutdown_connections 2024-01-21T20:02:47.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:47.421+0000 7f3b95bc2700 1 -- 172.21.15.146:0/44559673 wait complete. 2024-01-21T20:02:47.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:47.422+0000 7f3b95bc2700 1 Processor -- start 2024-01-21T20:02:47.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:47.422+0000 7f3b95bc2700 1 -- start start 2024-01-21T20:02:47.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:47.422+0000 7f3b95bc2700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b900f6510 0x7f3b9006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:47.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:47.422+0000 7f3b95bc2700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b90069e20 con 0x7f3b900f6510 2024-01-21T20:02:47.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:47.422+0000 7f3b8f7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b900f6510 0x7f3b9006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:47.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:47.423+0000 7f3b8f7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b900f6510 0x7f3b9006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:42326/0 (socket says 172.21.15.146:42326) 2024-01-21T20:02:47.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:47.423+0000 7f3b8f7fe700 1 -- 172.21.15.146:0/191758467 learned_addr learned my addr 172.21.15.146:0/191758467 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:02:47.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:47.423+0000 7f3b8f7fe700 1 -- 172.21.15.146:0/191758467 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b78005040 con 0x7f3b900f6510 2024-01-21T20:02:47.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:47.423+0000 7f3b8f7fe700 1 --2- 172.21.15.146:0/191758467 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b900f6510 0x7f3b9006b9a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f3b78013510 tx=0x7f3b7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:47.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:47.423+0000 7f3b87fff700 1 -- 172.21.15.146:0/191758467 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b78009320 con 0x7f3b900f6510 2024-01-21T20:02:47.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:47.423+0000 7f3b95bc2700 1 -- 172.21.15.146:0/191758467 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b90069fc0 con 0x7f3b900f6510 2024-01-21T20:02:47.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:47.423+0000 7f3b87fff700 1 -- 172.21.15.146:0/191758467 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3b78013f00 con 0x7f3b900f6510 2024-01-21T20:02:47.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:47.423+0000 7f3b87fff700 1 -- 172.21.15.146:0/191758467 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b78014d90 con 0x7f3b900f6510 2024-01-21T20:02:47.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:47.423+0000 7f3b95bc2700 1 -- 172.21.15.146:0/191758467 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b9006a450 con 0x7f3b900f6510 2024-01-21T20:02:47.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:47.424+0000 7f3b87fff700 1 -- 172.21.15.146:0/191758467 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3b78014420 con 0x7f3b900f6510 2024-01-21T20:02:47.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:47.424+0000 7f3b95bc2700 1 -- 172.21.15.146:0/191758467 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b90059100 con 0x7f3b900f6510 2024-01-21T20:02:47.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:47.424+0000 7f3b87fff700 1 --2- 172.21.15.146:0/191758467 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f3b7c02ff60 0x7f3b7c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:47.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:47.424+0000 7f3b87fff700 1 -- 172.21.15.146:0/191758467 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3b78041000 con 0x7f3b900f6510 2024-01-21T20:02:47.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:47.424+0000 7f3b8effd700 1 --2- 172.21.15.146:0/191758467 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f3b7c02ff60 0x7f3b7c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:47.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:47.425+0000 7f3b8effd700 1 --2- 172.21.15.146:0/191758467 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f3b7c02ff60 0x7f3b7c032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f3b800096a0 tx=0x7f3b80006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:47.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:47.429+0000 7f3b87fff700 1 -- 172.21.15.146:0/191758467 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3b780095f0 con 0x7f3b900f6510 2024-01-21T20:02:47.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:47.634+0000 7f3b95bc2700 1 -- 172.21.15.146:0/191758467 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3b900fbf60 con 0x7f3b900f6510 2024-01-21T20:02:47.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:47.634+0000 7f3b87fff700 1 -- 172.21.15.146:0/191758467 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3b7801db70 con 0x7f3b900f6510 2024-01-21T20:02:47.626 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:02:47.626 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:02:47.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:47.636+0000 7f3b95bc2700 1 -- 172.21.15.146:0/191758467 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f3b7c02ff60 msgr2=0x7f3b7c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:47.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:47.636+0000 7f3b95bc2700 1 --2- 172.21.15.146:0/191758467 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f3b7c02ff60 0x7f3b7c032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f3b800096a0 tx=0x7f3b80006b40).stop 2024-01-21T20:02:47.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:47.636+0000 7f3b95bc2700 1 -- 172.21.15.146:0/191758467 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b900f6510 msgr2=0x7f3b9006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:47.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:47.636+0000 7f3b95bc2700 1 --2- 172.21.15.146:0/191758467 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b900f6510 0x7f3b9006b9a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f3b78013510 tx=0x7f3b7800b040).stop 2024-01-21T20:02:47.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:47.636+0000 7f3b95bc2700 1 -- 172.21.15.146:0/191758467 shutdown_connections 2024-01-21T20:02:47.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:47.636+0000 7f3b95bc2700 1 --2- 172.21.15.146:0/191758467 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f3b7c02ff60 0x7f3b7c032420 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:47.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:47.636+0000 7f3b95bc2700 1 --2- 172.21.15.146:0/191758467 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b900f6510 0x7f3b9006b9a0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:47.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:47.636+0000 7f3b95bc2700 1 -- 172.21.15.146:0/191758467 >> 172.21.15.146:0/191758467 conn(0x7f3b900f1c70 msgr2=0x7f3b900f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:47.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:47.636+0000 7f3b95bc2700 1 -- 172.21.15.146:0/191758467 shutdown_connections 2024-01-21T20:02:47.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:47.636+0000 7f3b95bc2700 1 -- 172.21.15.146:0/191758467 wait complete. 2024-01-21T20:02:47.628 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:02:47.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:47 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:02:47.635619+0000 mon.smithi053 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.146:0/191758467' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:02:48.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:48 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:02:47.159855+0000 mgr.smithi053.fgsmol (mgr.14164) 110 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:49.066 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:02:49.067 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:02:50.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:50.272+0000 7fe00081a700 1 -- 172.21.15.146:0/2175092844 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdffc0f51a0 msgr2=0x7fdffc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:50.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:50.272+0000 7fe00081a700 1 --2- 172.21.15.146:0/2175092844 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdffc0f51a0 0x7fdffc0f55c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7fdfec009fb0 tx=0x7fdfec009b30).stop 2024-01-21T20:02:50.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:50.272+0000 7fe00081a700 1 -- 172.21.15.146:0/2175092844 shutdown_connections 2024-01-21T20:02:50.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:50.272+0000 7fe00081a700 1 --2- 172.21.15.146:0/2175092844 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdffc0f51a0 0x7fdffc0f55c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:50.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:50.272+0000 7fe00081a700 1 -- 172.21.15.146:0/2175092844 >> 172.21.15.146:0/2175092844 conn(0x7fdffc0f09a0 msgr2=0x7fdffc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:50.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:50.272+0000 7fe00081a700 1 -- 172.21.15.146:0/2175092844 shutdown_connections 2024-01-21T20:02:50.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:50.273+0000 7fe00081a700 1 -- 172.21.15.146:0/2175092844 wait complete. 2024-01-21T20:02:50.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:50.273+0000 7fe00081a700 1 Processor -- start 2024-01-21T20:02:50.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:50.273+0000 7fe00081a700 1 -- start start 2024-01-21T20:02:50.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:50.274+0000 7fe00081a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdffc0f51a0 0x7fdffc1091e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:50.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:50.274+0000 7fe00081a700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdffc060240 con 0x7fdffc0f51a0 2024-01-21T20:02:50.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:50.274+0000 7fdffb7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdffc0f51a0 0x7fdffc1091e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:50.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:50.274+0000 7fdffb7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdffc0f51a0 0x7fdffc1091e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:42354/0 (socket says 172.21.15.146:42354) 2024-01-21T20:02:50.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:50.274+0000 7fdffb7fe700 1 -- 172.21.15.146:0/1706363549 learned_addr learned my addr 172.21.15.146:0/1706363549 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:02:50.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:50.274+0000 7fdffb7fe700 1 -- 172.21.15.146:0/1706363549 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfec005040 con 0x7fdffc0f51a0 2024-01-21T20:02:50.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:50.274+0000 7fdffb7fe700 1 --2- 172.21.15.146:0/1706363549 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdffc0f51a0 0x7fdffc1091e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fdfec013510 tx=0x7fdfec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:50.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:50.274+0000 7fdfe3fff700 1 -- 172.21.15.146:0/1706363549 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfec009320 con 0x7fdffc0f51a0 2024-01-21T20:02:50.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:50.274+0000 7fe00081a700 1 -- 172.21.15.146:0/1706363549 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdffc060440 con 0x7fdffc0f51a0 2024-01-21T20:02:50.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:50.274+0000 7fdfe3fff700 1 -- 172.21.15.146:0/1706363549 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdfec013f00 con 0x7fdffc0f51a0 2024-01-21T20:02:50.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:50.275+0000 7fdfe3fff700 1 -- 172.21.15.146:0/1706363549 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfec014dc0 con 0x7fdffc0f51a0 2024-01-21T20:02:50.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:50.275+0000 7fe00081a700 1 -- 172.21.15.146:0/1706363549 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdffc109890 con 0x7fdffc0f51a0 2024-01-21T20:02:50.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:50.275+0000 7fdfe3fff700 1 -- 172.21.15.146:0/1706363549 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdfec0263a0 con 0x7fdffc0f51a0 2024-01-21T20:02:50.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:50.275+0000 7fdfe3fff700 1 --2- 172.21.15.146:0/1706363549 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fdfe402ffb0 0x7fdfe4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:50.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:50.275+0000 7fdfe3fff700 1 -- 172.21.15.146:0/1706363549 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdfec040ff0 con 0x7fdffc0f51a0 2024-01-21T20:02:50.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:50.276+0000 7fe00081a700 1 -- 172.21.15.146:0/1706363549 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdffc059100 con 0x7fdffc0f51a0 2024-01-21T20:02:50.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:50.276+0000 7fdffaffd700 1 --2- 172.21.15.146:0/1706363549 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fdfe402ffb0 0x7fdfe4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:50.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:50.276+0000 7fdffaffd700 1 --2- 172.21.15.146:0/1706363549 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fdfe402ffb0 0x7fdfe4032470 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fdff00096a0 tx=0x7fdff0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:50.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:50.280+0000 7fdfe3fff700 1 -- 172.21.15.146:0/1706363549 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdfec012920 con 0x7fdffc0f51a0 2024-01-21T20:02:50.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:50.484+0000 7fe00081a700 1 -- 172.21.15.146:0/1706363549 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdffc045ad0 con 0x7fdffc0f51a0 2024-01-21T20:02:50.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:50.485+0000 7fdfe3fff700 1 -- 172.21.15.146:0/1706363549 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdfec02b020 con 0x7fdffc0f51a0 2024-01-21T20:02:50.476 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:02:50.476 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:02:50.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:50.486+0000 7fe00081a700 1 -- 172.21.15.146:0/1706363549 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fdfe402ffb0 msgr2=0x7fdfe4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:50.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:50.486+0000 7fe00081a700 1 --2- 172.21.15.146:0/1706363549 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fdfe402ffb0 0x7fdfe4032470 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fdff00096a0 tx=0x7fdff0006b40).stop 2024-01-21T20:02:50.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:50.486+0000 7fe00081a700 1 -- 172.21.15.146:0/1706363549 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdffc0f51a0 msgr2=0x7fdffc1091e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:50.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:50.486+0000 7fe00081a700 1 --2- 172.21.15.146:0/1706363549 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdffc0f51a0 0x7fdffc1091e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fdfec013510 tx=0x7fdfec00b040).stop 2024-01-21T20:02:50.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:50.486+0000 7fe00081a700 1 -- 172.21.15.146:0/1706363549 shutdown_connections 2024-01-21T20:02:50.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:50.486+0000 7fe00081a700 1 --2- 172.21.15.146:0/1706363549 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fdfe402ffb0 0x7fdfe4032470 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:50.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:50.486+0000 7fe00081a700 1 --2- 172.21.15.146:0/1706363549 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdffc0f51a0 0x7fdffc1091e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:50.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:50.486+0000 7fe00081a700 1 -- 172.21.15.146:0/1706363549 >> 172.21.15.146:0/1706363549 conn(0x7fdffc0f09a0 msgr2=0x7fdffc0f13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:50.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:50.487+0000 7fe00081a700 1 -- 172.21.15.146:0/1706363549 shutdown_connections 2024-01-21T20:02:50.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:50.487+0000 7fe00081a700 1 -- 172.21.15.146:0/1706363549 wait complete. 2024-01-21T20:02:50.479 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:02:50.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:50 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:02:49.160098+0000 mgr.smithi053.fgsmol (mgr.14164) 111 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:50.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:50 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:02:50.486241+0000 mon.smithi053 (mon.0) 190 : audit [DBG] from='client.? 172.21.15.146:0/1706363549' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:02:51.925 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:02:51.925 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:02:52.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:52 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:02:51.160385+0000 mgr.smithi053.fgsmol (mgr.14164) 112 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:53.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:53.156+0000 7fa2c647a700 1 -- 172.21.15.146:0/1262647545 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2c00f6510 msgr2=0x7fa2c00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:53.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:53.156+0000 7fa2c647a700 1 --2- 172.21.15.146:0/1262647545 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2c00f6510 0x7fa2c00f6930 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fa2b0009fb0 tx=0x7fa2b0009b30).stop 2024-01-21T20:02:53.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:53.156+0000 7fa2c647a700 1 -- 172.21.15.146:0/1262647545 shutdown_connections 2024-01-21T20:02:53.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:53.156+0000 7fa2c647a700 1 --2- 172.21.15.146:0/1262647545 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2c00f6510 0x7fa2c00f6930 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:53.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:53.156+0000 7fa2c647a700 1 -- 172.21.15.146:0/1262647545 >> 172.21.15.146:0/1262647545 conn(0x7fa2c00f1c70 msgr2=0x7fa2c00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:53.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:53.157+0000 7fa2c647a700 1 -- 172.21.15.146:0/1262647545 shutdown_connections 2024-01-21T20:02:53.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:53.157+0000 7fa2c647a700 1 -- 172.21.15.146:0/1262647545 wait complete. 2024-01-21T20:02:53.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:53.157+0000 7fa2c647a700 1 Processor -- start 2024-01-21T20:02:53.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:53.157+0000 7fa2c647a700 1 -- start start 2024-01-21T20:02:53.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:53.157+0000 7fa2c647a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2c00f6510 0x7fa2c0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:53.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:53.157+0000 7fa2c647a700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c0106c20 con 0x7fa2c00f6510 2024-01-21T20:02:53.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:53.158+0000 7fa2bffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2c00f6510 0x7fa2c0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:53.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:53.158+0000 7fa2bffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2c00f6510 0x7fa2c0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:42376/0 (socket says 172.21.15.146:42376) 2024-01-21T20:02:53.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:53.158+0000 7fa2bffff700 1 -- 172.21.15.146:0/1078980607 learned_addr learned my addr 172.21.15.146:0/1078980607 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:02:53.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:53.158+0000 7fa2bffff700 1 -- 172.21.15.146:0/1078980607 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b0005040 con 0x7fa2c00f6510 2024-01-21T20:02:53.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:53.158+0000 7fa2bffff700 1 --2- 172.21.15.146:0/1078980607 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2c00f6510 0x7fa2c0106730 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fa2b0013510 tx=0x7fa2b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:53.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:53.158+0000 7fa2bcff9700 1 -- 172.21.15.146:0/1078980607 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2b0009320 con 0x7fa2c00f6510 2024-01-21T20:02:53.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:53.158+0000 7fa2c647a700 1 -- 172.21.15.146:0/1078980607 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2c0106e20 con 0x7fa2c00f6510 2024-01-21T20:02:53.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:53.158+0000 7fa2bcff9700 1 -- 172.21.15.146:0/1078980607 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa2b0013f00 con 0x7fa2c00f6510 2024-01-21T20:02:53.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:53.158+0000 7fa2bcff9700 1 -- 172.21.15.146:0/1078980607 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2b0014d90 con 0x7fa2c00f6510 2024-01-21T20:02:53.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:53.158+0000 7fa2c647a700 1 -- 172.21.15.146:0/1078980607 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2c01072b0 con 0x7fa2c00f6510 2024-01-21T20:02:53.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:53.159+0000 7fa2bcff9700 1 -- 172.21.15.146:0/1078980607 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa2b00095f0 con 0x7fa2c00f6510 2024-01-21T20:02:53.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:53.159+0000 7fa2c647a700 1 -- 172.21.15.146:0/1078980607 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2c0045ad0 con 0x7fa2c00f6510 2024-01-21T20:02:53.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:53.159+0000 7fa2bcff9700 1 --2- 172.21.15.146:0/1078980607 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa2a802ff60 0x7fa2a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:53.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:53.159+0000 7fa2bcff9700 1 -- 172.21.15.146:0/1078980607 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa2b00410c0 con 0x7fa2c00f6510 2024-01-21T20:02:53.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:53.159+0000 7fa2bf7fe700 1 --2- 172.21.15.146:0/1078980607 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa2a802ff60 0x7fa2a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:53.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:53.160+0000 7fa2bf7fe700 1 --2- 172.21.15.146:0/1078980607 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa2a802ff60 0x7fa2a8032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fa2b40096a0 tx=0x7fa2b4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:53.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:53.164+0000 7fa2bcff9700 1 -- 172.21.15.146:0/1078980607 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa2b0014420 con 0x7fa2c00f6510 2024-01-21T20:02:53.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:53.369+0000 7fa2c647a700 1 -- 172.21.15.146:0/1078980607 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa2c00fbf60 con 0x7fa2c00f6510 2024-01-21T20:02:53.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:53.369+0000 7fa2bcff9700 1 -- 172.21.15.146:0/1078980607 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa2b001d7f0 con 0x7fa2c00f6510 2024-01-21T20:02:53.363 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:02:53.363 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:02:53.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:53.371+0000 7fa2c647a700 1 -- 172.21.15.146:0/1078980607 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa2a802ff60 msgr2=0x7fa2a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:53.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:53.371+0000 7fa2c647a700 1 --2- 172.21.15.146:0/1078980607 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa2a802ff60 0x7fa2a8032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fa2b40096a0 tx=0x7fa2b4006b40).stop 2024-01-21T20:02:53.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:53.371+0000 7fa2c647a700 1 -- 172.21.15.146:0/1078980607 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2c00f6510 msgr2=0x7fa2c0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:53.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:53.371+0000 7fa2c647a700 1 --2- 172.21.15.146:0/1078980607 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2c00f6510 0x7fa2c0106730 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fa2b0013510 tx=0x7fa2b000b040).stop 2024-01-21T20:02:53.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:53.371+0000 7fa2c647a700 1 -- 172.21.15.146:0/1078980607 shutdown_connections 2024-01-21T20:02:53.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:53.371+0000 7fa2c647a700 1 --2- 172.21.15.146:0/1078980607 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa2a802ff60 0x7fa2a8032420 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:53.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:53.371+0000 7fa2c647a700 1 --2- 172.21.15.146:0/1078980607 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2c00f6510 0x7fa2c0106730 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:53.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:53.371+0000 7fa2c647a700 1 -- 172.21.15.146:0/1078980607 >> 172.21.15.146:0/1078980607 conn(0x7fa2c00f1c70 msgr2=0x7fa2c00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:53.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:53.372+0000 7fa2c647a700 1 -- 172.21.15.146:0/1078980607 shutdown_connections 2024-01-21T20:02:53.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:53.372+0000 7fa2c647a700 1 -- 172.21.15.146:0/1078980607 wait complete. 2024-01-21T20:02:53.365 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:02:53.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:53 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:02:53.370676+0000 mon.smithi053 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.146:0/1078980607' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:02:54.838 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:02:54.839 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:02:54.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:54 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:02:53.160583+0000 mgr.smithi053.fgsmol (mgr.14164) 113 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:56.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:56.054+0000 7fda790b5700 1 -- 172.21.15.146:0/1697427588 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda740f59d0 msgr2=0x7fda740f5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:56.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:56.054+0000 7fda790b5700 1 --2- 172.21.15.146:0/1697427588 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda740f59d0 0x7fda740f5df0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7fda6c009960 tx=0x7fda6c004d10).stop 2024-01-21T20:02:56.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:56.054+0000 7fda790b5700 1 -- 172.21.15.146:0/1697427588 shutdown_connections 2024-01-21T20:02:56.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:56.054+0000 7fda790b5700 1 --2- 172.21.15.146:0/1697427588 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda740f59d0 0x7fda740f5df0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:56.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:56.054+0000 7fda790b5700 1 -- 172.21.15.146:0/1697427588 >> 172.21.15.146:0/1697427588 conn(0x7fda740f02e0 msgr2=0x7fda740f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:56.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:56.054+0000 7fda790b5700 1 -- 172.21.15.146:0/1697427588 shutdown_connections 2024-01-21T20:02:56.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:56.054+0000 7fda790b5700 1 -- 172.21.15.146:0/1697427588 wait complete. 2024-01-21T20:02:56.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:56.055+0000 7fda790b5700 1 Processor -- start 2024-01-21T20:02:56.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:56.055+0000 7fda790b5700 1 -- start start 2024-01-21T20:02:56.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:56.055+0000 7fda790b5700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda740f59d0 0x7fda741093a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:56.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:56.055+0000 7fda790b5700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda740f53b0 con 0x7fda740f59d0 2024-01-21T20:02:56.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:56.055+0000 7fda72d9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda740f59d0 0x7fda741093a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:56.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:56.055+0000 7fda72d9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda740f59d0 0x7fda741093a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:42394/0 (socket says 172.21.15.146:42394) 2024-01-21T20:02:56.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:56.055+0000 7fda72d9d700 1 -- 172.21.15.146:0/47819045 learned_addr learned my addr 172.21.15.146:0/47819045 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:02:56.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:56.055+0000 7fda72d9d700 1 -- 172.21.15.146:0/47819045 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda6c009640 con 0x7fda740f59d0 2024-01-21T20:02:56.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:56.055+0000 7fda72d9d700 1 --2- 172.21.15.146:0/47819045 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda740f59d0 0x7fda741093a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7fda6c004b70 tx=0x7fda6c004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:56.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:56.056+0000 7fda5b7fe700 1 -- 172.21.15.146:0/47819045 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda6c018070 con 0x7fda740f59d0 2024-01-21T20:02:56.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:56.056+0000 7fda790b5700 1 -- 172.21.15.146:0/47819045 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda740f55b0 con 0x7fda740f59d0 2024-01-21T20:02:56.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:56.056+0000 7fda5b7fe700 1 -- 172.21.15.146:0/47819045 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fda6c00b870 con 0x7fda740f59d0 2024-01-21T20:02:56.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:56.056+0000 7fda5b7fe700 1 -- 172.21.15.146:0/47819045 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda6c01c560 con 0x7fda740f59d0 2024-01-21T20:02:56.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:56.056+0000 7fda790b5700 1 -- 172.21.15.146:0/47819045 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda74109a50 con 0x7fda740f59d0 2024-01-21T20:02:56.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:56.056+0000 7fda5b7fe700 1 -- 172.21.15.146:0/47819045 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fda6c013630 con 0x7fda740f59d0 2024-01-21T20:02:56.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:56.056+0000 7fda790b5700 1 -- 172.21.15.146:0/47819045 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda74045ad0 con 0x7fda740f59d0 2024-01-21T20:02:56.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:56.057+0000 7fda5b7fe700 1 --2- 172.21.15.146:0/47819045 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fda5c02ff60 0x7fda5c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:56.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:56.057+0000 7fda5b7fe700 1 -- 172.21.15.146:0/47819045 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fda6c041450 con 0x7fda740f59d0 2024-01-21T20:02:56.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:56.057+0000 7fda7259c700 1 --2- 172.21.15.146:0/47819045 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fda5c02ff60 0x7fda5c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:56.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:56.057+0000 7fda7259c700 1 --2- 172.21.15.146:0/47819045 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fda5c02ff60 0x7fda5c032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fda680096a0 tx=0x7fda68006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:56.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:56.061+0000 7fda5b7fe700 1 -- 172.21.15.146:0/47819045 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fda6c004810 con 0x7fda740f59d0 2024-01-21T20:02:56.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:56.267+0000 7fda790b5700 1 -- 172.21.15.146:0/47819045 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fda7410a270 con 0x7fda740f59d0 2024-01-21T20:02:56.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:56.268+0000 7fda5b7fe700 1 -- 172.21.15.146:0/47819045 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fda6c012380 con 0x7fda740f59d0 2024-01-21T20:02:56.259 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:02:56.260 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:02:56.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:56.270+0000 7fda790b5700 1 -- 172.21.15.146:0/47819045 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fda5c02ff60 msgr2=0x7fda5c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:56.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:56.270+0000 7fda790b5700 1 --2- 172.21.15.146:0/47819045 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fda5c02ff60 0x7fda5c032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fda680096a0 tx=0x7fda68006b40).stop 2024-01-21T20:02:56.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:56.270+0000 7fda790b5700 1 -- 172.21.15.146:0/47819045 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda740f59d0 msgr2=0x7fda741093a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:56.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:56.270+0000 7fda790b5700 1 --2- 172.21.15.146:0/47819045 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda740f59d0 0x7fda741093a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7fda6c004b70 tx=0x7fda6c004d10).stop 2024-01-21T20:02:56.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:56.270+0000 7fda790b5700 1 -- 172.21.15.146:0/47819045 shutdown_connections 2024-01-21T20:02:56.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:56.270+0000 7fda790b5700 1 --2- 172.21.15.146:0/47819045 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fda5c02ff60 0x7fda5c032420 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:56.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:56.270+0000 7fda790b5700 1 --2- 172.21.15.146:0/47819045 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda740f59d0 0x7fda741093a0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:56.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:56.270+0000 7fda790b5700 1 -- 172.21.15.146:0/47819045 >> 172.21.15.146:0/47819045 conn(0x7fda740f02e0 msgr2=0x7fda740f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:56.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:56.270+0000 7fda790b5700 1 -- 172.21.15.146:0/47819045 shutdown_connections 2024-01-21T20:02:56.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:56.270+0000 7fda790b5700 1 -- 172.21.15.146:0/47819045 wait complete. 2024-01-21T20:02:56.262 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:02:56.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:56 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:02:55.160924+0000 mgr.smithi053.fgsmol (mgr.14164) 114 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:56.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:56 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:02:56.269456+0000 mon.smithi053 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.146:0/47819045' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:02:57.651 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:02:57.652 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:02:58.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:58.886+0000 7f2e6933b700 1 -- 172.21.15.146:0/2861724091 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e640f5410 msgr2=0x7f2e640f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:58.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:58.886+0000 7f2e6933b700 1 --2- 172.21.15.146:0/2861724091 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e640f5410 0x7f2e640f5830 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f2e58009fb0 tx=0x7f2e58009b30).stop 2024-01-21T20:02:58.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:58.886+0000 7f2e6933b700 1 -- 172.21.15.146:0/2861724091 shutdown_connections 2024-01-21T20:02:58.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:58.886+0000 7f2e6933b700 1 --2- 172.21.15.146:0/2861724091 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e640f5410 0x7f2e640f5830 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:58.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:58.886+0000 7f2e6933b700 1 -- 172.21.15.146:0/2861724091 >> 172.21.15.146:0/2861724091 conn(0x7f2e640f0b70 msgr2=0x7f2e640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:58.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:58.886+0000 7f2e6933b700 1 -- 172.21.15.146:0/2861724091 shutdown_connections 2024-01-21T20:02:58.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:58.886+0000 7f2e6933b700 1 -- 172.21.15.146:0/2861724091 wait complete. 2024-01-21T20:02:58.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:58.887+0000 7f2e6933b700 1 Processor -- start 2024-01-21T20:02:58.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:58.887+0000 7f2e6933b700 1 -- start start 2024-01-21T20:02:58.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:58.887+0000 7f2e6933b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e640f5410 0x7f2e641055d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:58.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:58.887+0000 7f2e6933b700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e64105ac0 con 0x7f2e640f5410 2024-01-21T20:02:58.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:58.887+0000 7f2e62ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e640f5410 0x7f2e641055d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:58.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:58.887+0000 7f2e62ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e640f5410 0x7f2e641055d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:54478/0 (socket says 172.21.15.146:54478) 2024-01-21T20:02:58.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:58.887+0000 7f2e62ffd700 1 -- 172.21.15.146:0/774788915 learned_addr learned my addr 172.21.15.146:0/774788915 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:02:58.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:58.888+0000 7f2e62ffd700 1 -- 172.21.15.146:0/774788915 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e58005040 con 0x7f2e640f5410 2024-01-21T20:02:58.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:58.888+0000 7f2e62ffd700 1 --2- 172.21.15.146:0/774788915 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e640f5410 0x7f2e641055d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f2e58013510 tx=0x7f2e5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:58.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:58.888+0000 7f2e4b7fe700 1 -- 172.21.15.146:0/774788915 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e58009320 con 0x7f2e640f5410 2024-01-21T20:02:58.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:58.888+0000 7f2e6933b700 1 -- 172.21.15.146:0/774788915 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e64105cc0 con 0x7f2e640f5410 2024-01-21T20:02:58.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:58.888+0000 7f2e4b7fe700 1 -- 172.21.15.146:0/774788915 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2e58013f00 con 0x7f2e640f5410 2024-01-21T20:02:58.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:58.888+0000 7f2e4b7fe700 1 -- 172.21.15.146:0/774788915 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e58014d90 con 0x7f2e640f5410 2024-01-21T20:02:58.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:58.888+0000 7f2e6933b700 1 -- 172.21.15.146:0/774788915 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e64106150 con 0x7f2e640f5410 2024-01-21T20:02:58.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:58.889+0000 7f2e4b7fe700 1 -- 172.21.15.146:0/774788915 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2e580263a0 con 0x7f2e640f5410 2024-01-21T20:02:58.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:58.889+0000 7f2e6933b700 1 -- 172.21.15.146:0/774788915 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e64045ad0 con 0x7f2e640f5410 2024-01-21T20:02:58.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:58.889+0000 7f2e4b7fe700 1 --2- 172.21.15.146:0/774788915 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f2e4c02ff60 0x7f2e4c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:58.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:58.889+0000 7f2e4b7fe700 1 -- 172.21.15.146:0/774788915 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2e58041380 con 0x7f2e640f5410 2024-01-21T20:02:58.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:58.889+0000 7f2e627fc700 1 --2- 172.21.15.146:0/774788915 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f2e4c02ff60 0x7f2e4c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:58.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:58.890+0000 7f2e627fc700 1 --2- 172.21.15.146:0/774788915 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f2e4c02ff60 0x7f2e4c032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f2e540096a0 tx=0x7f2e54006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:58.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:58.894+0000 7f2e4b7fe700 1 -- 172.21.15.146:0/774788915 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2e5801cb50 con 0x7f2e640f5410 2024-01-21T20:02:58.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:58 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:02:57.161186+0000 mgr.smithi053.fgsmol (mgr.14164) 115 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:59.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:59.098+0000 7f2e6933b700 1 -- 172.21.15.146:0/774788915 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2e64107000 con 0x7f2e640f5410 2024-01-21T20:02:59.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:59.099+0000 7f2e4b7fe700 1 -- 172.21.15.146:0/774788915 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2e5801c970 con 0x7f2e640f5410 2024-01-21T20:02:59.090 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:02:59.090 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:02:59.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:59.101+0000 7f2e6933b700 1 -- 172.21.15.146:0/774788915 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f2e4c02ff60 msgr2=0x7f2e4c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:59.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:59.101+0000 7f2e6933b700 1 --2- 172.21.15.146:0/774788915 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f2e4c02ff60 0x7f2e4c032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f2e540096a0 tx=0x7f2e54006b40).stop 2024-01-21T20:02:59.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:59.101+0000 7f2e6933b700 1 -- 172.21.15.146:0/774788915 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e640f5410 msgr2=0x7f2e641055d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:59.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:59.101+0000 7f2e6933b700 1 --2- 172.21.15.146:0/774788915 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e640f5410 0x7f2e641055d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f2e58013510 tx=0x7f2e5800b040).stop 2024-01-21T20:02:59.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:59.101+0000 7f2e6933b700 1 -- 172.21.15.146:0/774788915 shutdown_connections 2024-01-21T20:02:59.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:59.101+0000 7f2e6933b700 1 --2- 172.21.15.146:0/774788915 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f2e4c02ff60 0x7f2e4c032420 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:59.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:59.101+0000 7f2e6933b700 1 --2- 172.21.15.146:0/774788915 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e640f5410 0x7f2e641055d0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:59.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:59.101+0000 7f2e6933b700 1 -- 172.21.15.146:0/774788915 >> 172.21.15.146:0/774788915 conn(0x7f2e640f0b70 msgr2=0x7f2e640f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:59.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:59.101+0000 7f2e6933b700 1 -- 172.21.15.146:0/774788915 shutdown_connections 2024-01-21T20:02:59.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:02:59.101+0000 7f2e6933b700 1 -- 172.21.15.146:0/774788915 wait complete. 2024-01-21T20:02:59.093 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:02:59.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:02:59 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:02:59.100475+0000 mon.smithi053 (mon.0) 193 : audit [DBG] from='client.? 172.21.15.146:0/774788915' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:03:00.545 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:03:00.546 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:03:00.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:00 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:02:59.161470+0000 mgr.smithi053.fgsmol (mgr.14164) 116 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:01.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:01.745+0000 7f638e96c700 1 -- 172.21.15.146:0/1666903185 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63880f6510 msgr2=0x7f63880f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:01.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:01.745+0000 7f638e96c700 1 --2- 172.21.15.146:0/1666903185 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63880f6510 0x7f63880f6930 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f6378009fb0 tx=0x7f6378009b30).stop 2024-01-21T20:03:01.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:01.746+0000 7f638e96c700 1 -- 172.21.15.146:0/1666903185 shutdown_connections 2024-01-21T20:03:01.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:01.746+0000 7f638e96c700 1 --2- 172.21.15.146:0/1666903185 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63880f6510 0x7f63880f6930 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:01.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:01.746+0000 7f638e96c700 1 -- 172.21.15.146:0/1666903185 >> 172.21.15.146:0/1666903185 conn(0x7f63880f1c70 msgr2=0x7f63880f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:01.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:01.746+0000 7f638e96c700 1 -- 172.21.15.146:0/1666903185 shutdown_connections 2024-01-21T20:03:01.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:01.746+0000 7f638e96c700 1 -- 172.21.15.146:0/1666903185 wait complete. 2024-01-21T20:03:01.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:01.746+0000 7f638e96c700 1 Processor -- start 2024-01-21T20:03:01.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:01.746+0000 7f638e96c700 1 -- start start 2024-01-21T20:03:01.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:01.746+0000 7f638e96c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63880f6510 0x7f6388106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:01.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:01.746+0000 7f638e96c700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6388106c20 con 0x7f63880f6510 2024-01-21T20:03:01.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:01.747+0000 7f6387fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63880f6510 0x7f6388106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:01.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:01.747+0000 7f6387fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63880f6510 0x7f6388106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:54494/0 (socket says 172.21.15.146:54494) 2024-01-21T20:03:01.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:01.747+0000 7f6387fff700 1 -- 172.21.15.146:0/3417457304 learned_addr learned my addr 172.21.15.146:0/3417457304 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:03:01.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:01.747+0000 7f6387fff700 1 -- 172.21.15.146:0/3417457304 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6378005040 con 0x7f63880f6510 2024-01-21T20:03:01.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:01.747+0000 7f6387fff700 1 --2- 172.21.15.146:0/3417457304 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63880f6510 0x7f6388106730 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f6378013510 tx=0x7f637800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:01.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:01.747+0000 7f6384ff9700 1 -- 172.21.15.146:0/3417457304 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6378009320 con 0x7f63880f6510 2024-01-21T20:03:01.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:01.747+0000 7f638e96c700 1 -- 172.21.15.146:0/3417457304 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6388106e20 con 0x7f63880f6510 2024-01-21T20:03:01.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:01.747+0000 7f6384ff9700 1 -- 172.21.15.146:0/3417457304 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6378013f00 con 0x7f63880f6510 2024-01-21T20:03:01.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:01.747+0000 7f6384ff9700 1 -- 172.21.15.146:0/3417457304 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6378014d90 con 0x7f63880f6510 2024-01-21T20:03:01.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:01.747+0000 7f638e96c700 1 -- 172.21.15.146:0/3417457304 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63881072b0 con 0x7f63880f6510 2024-01-21T20:03:01.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:01.748+0000 7f6384ff9700 1 -- 172.21.15.146:0/3417457304 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f63780095f0 con 0x7f63880f6510 2024-01-21T20:03:01.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:01.748+0000 7f638e96c700 1 -- 172.21.15.146:0/3417457304 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6388045ad0 con 0x7f63880f6510 2024-01-21T20:03:01.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:01.748+0000 7f6384ff9700 1 --2- 172.21.15.146:0/3417457304 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f637002ff60 0x7f6370032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:01.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:01.748+0000 7f6384ff9700 1 -- 172.21.15.146:0/3417457304 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6378040fb0 con 0x7f63880f6510 2024-01-21T20:03:01.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:01.748+0000 7f63877fe700 1 --2- 172.21.15.146:0/3417457304 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f637002ff60 0x7f6370032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:01.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:01.749+0000 7f63877fe700 1 --2- 172.21.15.146:0/3417457304 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f637002ff60 0x7f6370032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f637c0096a0 tx=0x7f637c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:01.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:01.753+0000 7f6384ff9700 1 -- 172.21.15.146:0/3417457304 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6378014420 con 0x7f63880f6510 2024-01-21T20:03:01.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:01.957+0000 7f638e96c700 1 -- 172.21.15.146:0/3417457304 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f63880fbf60 con 0x7f63880f6510 2024-01-21T20:03:01.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:01.958+0000 7f6384ff9700 1 -- 172.21.15.146:0/3417457304 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6378010230 con 0x7f63880f6510 2024-01-21T20:03:01.949 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:03:01.949 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:03:01.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:01.959+0000 7f638e96c700 1 -- 172.21.15.146:0/3417457304 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f637002ff60 msgr2=0x7f6370032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:01.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:01.959+0000 7f638e96c700 1 --2- 172.21.15.146:0/3417457304 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f637002ff60 0x7f6370032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f637c0096a0 tx=0x7f637c006b40).stop 2024-01-21T20:03:01.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:01.959+0000 7f638e96c700 1 -- 172.21.15.146:0/3417457304 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63880f6510 msgr2=0x7f6388106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:01.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:01.959+0000 7f638e96c700 1 --2- 172.21.15.146:0/3417457304 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63880f6510 0x7f6388106730 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f6378013510 tx=0x7f637800b040).stop 2024-01-21T20:03:01.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:01.959+0000 7f638e96c700 1 -- 172.21.15.146:0/3417457304 shutdown_connections 2024-01-21T20:03:01.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:01.959+0000 7f638e96c700 1 --2- 172.21.15.146:0/3417457304 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f637002ff60 0x7f6370032420 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:01.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:01.959+0000 7f638e96c700 1 --2- 172.21.15.146:0/3417457304 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63880f6510 0x7f6388106730 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:01.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:01.959+0000 7f638e96c700 1 -- 172.21.15.146:0/3417457304 >> 172.21.15.146:0/3417457304 conn(0x7f63880f1c70 msgr2=0x7f63880f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:01.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:01.959+0000 7f638e96c700 1 -- 172.21.15.146:0/3417457304 shutdown_connections 2024-01-21T20:03:01.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:01.959+0000 7f638e96c700 1 -- 172.21.15.146:0/3417457304 wait complete. 2024-01-21T20:03:01.951 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:03:02.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:02 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:03:01.161814+0000 mgr.smithi053.fgsmol (mgr.14164) 117 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:02.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:02 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:03:01.959020+0000 mon.smithi053 (mon.0) 194 : audit [DBG] from='client.? 172.21.15.146:0/3417457304' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:03:03.367 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:03:03.368 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:03:04.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:04.676+0000 7f5ca49b1700 1 -- 172.21.15.146:0/3334529309 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ca00f5410 msgr2=0x7f5ca00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:04.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:04.676+0000 7f5ca49b1700 1 --2- 172.21.15.146:0/3334529309 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ca00f5410 0x7f5ca00f5830 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f5c88009fb0 tx=0x7f5c88009b30).stop 2024-01-21T20:03:04.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:04.677+0000 7f5ca49b1700 1 -- 172.21.15.146:0/3334529309 shutdown_connections 2024-01-21T20:03:04.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:04.677+0000 7f5ca49b1700 1 --2- 172.21.15.146:0/3334529309 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ca00f5410 0x7f5ca00f5830 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:04.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:04.677+0000 7f5ca49b1700 1 -- 172.21.15.146:0/3334529309 >> 172.21.15.146:0/3334529309 conn(0x7f5ca00f0b70 msgr2=0x7f5ca00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:04.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:04.677+0000 7f5ca49b1700 1 -- 172.21.15.146:0/3334529309 shutdown_connections 2024-01-21T20:03:04.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:04.677+0000 7f5ca49b1700 1 -- 172.21.15.146:0/3334529309 wait complete. 2024-01-21T20:03:04.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:04.677+0000 7f5ca49b1700 1 Processor -- start 2024-01-21T20:03:04.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:04.677+0000 7f5ca49b1700 1 -- start start 2024-01-21T20:03:04.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:04.677+0000 7f5ca49b1700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ca00f5410 0x7f5ca0105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:04.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:04.678+0000 7f5ca49b1700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca0105b20 con 0x7f5ca00f5410 2024-01-21T20:03:04.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:04.678+0000 7f5c9e59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ca00f5410 0x7f5ca0105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:04.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:04.678+0000 7f5c9e59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ca00f5410 0x7f5ca0105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:54516/0 (socket says 172.21.15.146:54516) 2024-01-21T20:03:04.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:04.678+0000 7f5c9e59c700 1 -- 172.21.15.146:0/3974742993 learned_addr learned my addr 172.21.15.146:0/3974742993 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:03:04.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:04.678+0000 7f5c9e59c700 1 -- 172.21.15.146:0/3974742993 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c88005040 con 0x7f5ca00f5410 2024-01-21T20:03:04.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:04.678+0000 7f5c9e59c700 1 --2- 172.21.15.146:0/3974742993 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ca00f5410 0x7f5ca0105630 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f5c88013510 tx=0x7f5c8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:04.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:04.678+0000 7f5c96ffd700 1 -- 172.21.15.146:0/3974742993 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c88009320 con 0x7f5ca00f5410 2024-01-21T20:03:04.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:04.678+0000 7f5ca49b1700 1 -- 172.21.15.146:0/3974742993 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ca0105d20 con 0x7f5ca00f5410 2024-01-21T20:03:04.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:04.678+0000 7f5c96ffd700 1 -- 172.21.15.146:0/3974742993 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5c88013f00 con 0x7f5ca00f5410 2024-01-21T20:03:04.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:04.678+0000 7f5c96ffd700 1 -- 172.21.15.146:0/3974742993 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c88014d90 con 0x7f5ca00f5410 2024-01-21T20:03:04.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:04.679+0000 7f5ca49b1700 1 -- 172.21.15.146:0/3974742993 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ca01061b0 con 0x7f5ca00f5410 2024-01-21T20:03:04.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:04.679+0000 7f5c96ffd700 1 -- 172.21.15.146:0/3974742993 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5c880095f0 con 0x7f5ca00f5410 2024-01-21T20:03:04.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:04.679+0000 7f5ca49b1700 1 -- 172.21.15.146:0/3974742993 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ca0045ad0 con 0x7f5ca00f5410 2024-01-21T20:03:04.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:04.680+0000 7f5c96ffd700 1 --2- 172.21.15.146:0/3974742993 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f5c8c02ff60 0x7f5c8c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:04.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:04.680+0000 7f5c96ffd700 1 -- 172.21.15.146:0/3974742993 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5c88040fe0 con 0x7f5ca00f5410 2024-01-21T20:03:04.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:04.680+0000 7f5c9dd9b700 1 --2- 172.21.15.146:0/3974742993 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f5c8c02ff60 0x7f5c8c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:04.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:04.680+0000 7f5c9dd9b700 1 --2- 172.21.15.146:0/3974742993 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f5c8c02ff60 0x7f5c8c032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f5c900096a0 tx=0x7f5c90006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:04.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:04.684+0000 7f5c96ffd700 1 -- 172.21.15.146:0/3974742993 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5c8801d7f0 con 0x7f5ca00f5410 2024-01-21T20:03:04.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:04.890+0000 7f5ca49b1700 1 -- 172.21.15.146:0/3974742993 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5ca00fae60 con 0x7f5ca00f5410 2024-01-21T20:03:04.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:04.891+0000 7f5c96ffd700 1 -- 172.21.15.146:0/3974742993 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5c88010230 con 0x7f5ca00f5410 2024-01-21T20:03:04.882 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:03:04.882 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:03:04.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:04.892+0000 7f5ca49b1700 1 -- 172.21.15.146:0/3974742993 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f5c8c02ff60 msgr2=0x7f5c8c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:04.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:04.892+0000 7f5ca49b1700 1 --2- 172.21.15.146:0/3974742993 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f5c8c02ff60 0x7f5c8c032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f5c900096a0 tx=0x7f5c90006b40).stop 2024-01-21T20:03:04.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:04.892+0000 7f5ca49b1700 1 -- 172.21.15.146:0/3974742993 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ca00f5410 msgr2=0x7f5ca0105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:04.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:04.892+0000 7f5ca49b1700 1 --2- 172.21.15.146:0/3974742993 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ca00f5410 0x7f5ca0105630 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f5c88013510 tx=0x7f5c8800b040).stop 2024-01-21T20:03:04.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:04.892+0000 7f5ca49b1700 1 -- 172.21.15.146:0/3974742993 shutdown_connections 2024-01-21T20:03:04.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:04.892+0000 7f5ca49b1700 1 --2- 172.21.15.146:0/3974742993 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f5c8c02ff60 0x7f5c8c032420 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:04.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:04.892+0000 7f5ca49b1700 1 --2- 172.21.15.146:0/3974742993 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ca00f5410 0x7f5ca0105630 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:04.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:04.892+0000 7f5ca49b1700 1 -- 172.21.15.146:0/3974742993 >> 172.21.15.146:0/3974742993 conn(0x7f5ca00f0b70 msgr2=0x7f5ca00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:04.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:04.892+0000 7f5ca49b1700 1 -- 172.21.15.146:0/3974742993 shutdown_connections 2024-01-21T20:03:04.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:04.893+0000 7f5ca49b1700 1 -- 172.21.15.146:0/3974742993 wait complete. 2024-01-21T20:03:04.884 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:03:04.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:04 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:03:03.162130+0000 mgr.smithi053.fgsmol (mgr.14164) 118 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:06.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:05 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:03:04.892023+0000 mon.smithi053 (mon.0) 195 : audit [DBG] from='client.? 172.21.15.146:0/3974742993' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:03:06.269 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:03:06.270 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:03:07.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:06 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:03:05.162416+0000 mgr.smithi053.fgsmol (mgr.14164) 119 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:07.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:07.434+0000 7f33620cf700 1 -- 172.21.15.146:0/2841871120 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f335c0f62a0 msgr2=0x7f335c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:07.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:07.434+0000 7f33620cf700 1 --2- 172.21.15.146:0/2841871120 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f335c0f62a0 0x7f335c0f66c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f334c009fb0 tx=0x7f334c009b30).stop 2024-01-21T20:03:07.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:07.434+0000 7f33620cf700 1 -- 172.21.15.146:0/2841871120 shutdown_connections 2024-01-21T20:03:07.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:07.434+0000 7f33620cf700 1 --2- 172.21.15.146:0/2841871120 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f335c0f62a0 0x7f335c0f66c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:07.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:07.434+0000 7f33620cf700 1 -- 172.21.15.146:0/2841871120 >> 172.21.15.146:0/2841871120 conn(0x7f335c0f1aa0 msgr2=0x7f335c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:07.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:07.434+0000 7f33620cf700 1 -- 172.21.15.146:0/2841871120 shutdown_connections 2024-01-21T20:03:07.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:07.434+0000 7f33620cf700 1 -- 172.21.15.146:0/2841871120 wait complete. 2024-01-21T20:03:07.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:07.434+0000 7f33620cf700 1 Processor -- start 2024-01-21T20:03:07.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:07.434+0000 7f33620cf700 1 -- start start 2024-01-21T20:03:07.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:07.434+0000 7f33620cf700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f335c0f62a0 0x7f335c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:07.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:07.434+0000 7f33620cf700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f335c069e20 con 0x7f335c0f62a0 2024-01-21T20:03:07.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:07.435+0000 7f33610cd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f335c0f62a0 0x7f335c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:07.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:07.435+0000 7f33610cd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f335c0f62a0 0x7f335c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:57436/0 (socket says 172.21.15.146:57436) 2024-01-21T20:03:07.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:07.435+0000 7f33610cd700 1 -- 172.21.15.146:0/4241543673 learned_addr learned my addr 172.21.15.146:0/4241543673 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:03:07.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:07.435+0000 7f33610cd700 1 -- 172.21.15.146:0/4241543673 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f334c005040 con 0x7f335c0f62a0 2024-01-21T20:03:07.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:07.435+0000 7f33610cd700 1 --2- 172.21.15.146:0/4241543673 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f335c0f62a0 0x7f335c06b9a0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f334c013510 tx=0x7f334c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:07.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:07.435+0000 7f3351ffb700 1 -- 172.21.15.146:0/4241543673 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f334c009320 con 0x7f335c0f62a0 2024-01-21T20:03:07.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:07.435+0000 7f33620cf700 1 -- 172.21.15.146:0/4241543673 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f335c069fc0 con 0x7f335c0f62a0 2024-01-21T20:03:07.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:07.435+0000 7f3351ffb700 1 -- 172.21.15.146:0/4241543673 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f334c013f00 con 0x7f335c0f62a0 2024-01-21T20:03:07.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:07.435+0000 7f3351ffb700 1 -- 172.21.15.146:0/4241543673 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f334c014dc0 con 0x7f335c0f62a0 2024-01-21T20:03:07.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:07.435+0000 7f33620cf700 1 -- 172.21.15.146:0/4241543673 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f335c06a450 con 0x7f335c0f62a0 2024-01-21T20:03:07.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:07.436+0000 7f33620cf700 1 -- 172.21.15.146:0/4241543673 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f335c059100 con 0x7f335c0f62a0 2024-01-21T20:03:07.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:07.436+0000 7f3351ffb700 1 -- 172.21.15.146:0/4241543673 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f334c0095f0 con 0x7f335c0f62a0 2024-01-21T20:03:07.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:07.436+0000 7f3351ffb700 1 --2- 172.21.15.146:0/4241543673 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f334802ff60 0x7f3348032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:07.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:07.437+0000 7f3351ffb700 1 -- 172.21.15.146:0/4241543673 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f334c041020 con 0x7f335c0f62a0 2024-01-21T20:03:07.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:07.437+0000 7f33608cc700 1 --2- 172.21.15.146:0/4241543673 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f334802ff60 0x7f3348032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:07.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:07.437+0000 7f33608cc700 1 --2- 172.21.15.146:0/4241543673 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f334802ff60 0x7f3348032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f33580096a0 tx=0x7f3358006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:07.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:07.441+0000 7f3351ffb700 1 -- 172.21.15.146:0/4241543673 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f334c014420 con 0x7f335c0f62a0 2024-01-21T20:03:07.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:07.648+0000 7f33620cf700 1 -- 172.21.15.146:0/4241543673 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f335c045ad0 con 0x7f335c0f62a0 2024-01-21T20:03:07.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:07.649+0000 7f3351ffb700 1 -- 172.21.15.146:0/4241543673 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f334c01c920 con 0x7f335c0f62a0 2024-01-21T20:03:07.640 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:03:07.640 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:03:07.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:07.651+0000 7f33620cf700 1 -- 172.21.15.146:0/4241543673 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f334802ff60 msgr2=0x7f3348032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:07.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:07.651+0000 7f33620cf700 1 --2- 172.21.15.146:0/4241543673 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f334802ff60 0x7f3348032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f33580096a0 tx=0x7f3358006b40).stop 2024-01-21T20:03:07.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:07.651+0000 7f33620cf700 1 -- 172.21.15.146:0/4241543673 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f335c0f62a0 msgr2=0x7f335c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:07.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:07.651+0000 7f33620cf700 1 --2- 172.21.15.146:0/4241543673 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f335c0f62a0 0x7f335c06b9a0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f334c013510 tx=0x7f334c00b040).stop 2024-01-21T20:03:07.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:07.651+0000 7f33620cf700 1 -- 172.21.15.146:0/4241543673 shutdown_connections 2024-01-21T20:03:07.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:07.651+0000 7f33620cf700 1 --2- 172.21.15.146:0/4241543673 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f334802ff60 0x7f3348032420 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:07.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:07.651+0000 7f33620cf700 1 --2- 172.21.15.146:0/4241543673 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f335c0f62a0 0x7f335c06b9a0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:07.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:07.651+0000 7f33620cf700 1 -- 172.21.15.146:0/4241543673 >> 172.21.15.146:0/4241543673 conn(0x7f335c0f1aa0 msgr2=0x7f335c0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:07.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:07.651+0000 7f33620cf700 1 -- 172.21.15.146:0/4241543673 shutdown_connections 2024-01-21T20:03:07.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:07.651+0000 7f33620cf700 1 -- 172.21.15.146:0/4241543673 wait complete. 2024-01-21T20:03:07.643 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:03:08.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:07 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:03:07.650314+0000 mon.smithi053 (mon.0) 196 : audit [DBG] from='client.? 172.21.15.146:0/4241543673' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:03:09.052 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:03:09.052 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:03:09.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:08 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:03:07.162751+0000 mgr.smithi053.fgsmol (mgr.14164) 120 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:10.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:09 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:03:09.266468+0000 mon.smithi053 (mon.0) 197 : audit [INF] from='mgr.14164 172.21.15.53:0/2331671680' entity='mgr.smithi053.fgsmol' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fgsmol/mirror_snapshot_schedule"}]: dispatch 2024-01-21T20:03:10.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:09 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:03:09.274682+0000 mon.smithi053 (mon.0) 198 : audit [INF] from='mgr.14164 172.21.15.53:0/2331671680' entity='mgr.smithi053.fgsmol' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fgsmol/trash_purge_schedule"}]: dispatch 2024-01-21T20:03:10.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:10.258+0000 7f64ce3ab700 1 -- 172.21.15.146:0/920285746 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64c8060280 msgr2=0x7f64c80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:10.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:10.258+0000 7f64ce3ab700 1 --2- 172.21.15.146:0/920285746 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64c8060280 0x7f64c80f90f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f64b8009fb0 tx=0x7f64b8009b30).stop 2024-01-21T20:03:10.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:10.258+0000 7f64ce3ab700 1 -- 172.21.15.146:0/920285746 shutdown_connections 2024-01-21T20:03:10.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:10.258+0000 7f64ce3ab700 1 --2- 172.21.15.146:0/920285746 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64c8060280 0x7f64c80f90f0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:10.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:10.258+0000 7f64ce3ab700 1 -- 172.21.15.146:0/920285746 >> 172.21.15.146:0/920285746 conn(0x7f64c80f0b70 msgr2=0x7f64c80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:10.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:10.258+0000 7f64ce3ab700 1 -- 172.21.15.146:0/920285746 shutdown_connections 2024-01-21T20:03:10.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:10.258+0000 7f64ce3ab700 1 -- 172.21.15.146:0/920285746 wait complete. 2024-01-21T20:03:10.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:10.259+0000 7f64ce3ab700 1 Processor -- start 2024-01-21T20:03:10.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:10.259+0000 7f64ce3ab700 1 -- start start 2024-01-21T20:03:10.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:10.259+0000 7f64ce3ab700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64c8060280 0x7f64c81066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:10.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:10.259+0000 7f64ce3ab700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64c8106bd0 con 0x7f64c8060280 2024-01-21T20:03:10.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:10.259+0000 7f64c7fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64c8060280 0x7f64c81066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:10.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:10.259+0000 7f64c7fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64c8060280 0x7f64c81066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:57458/0 (socket says 172.21.15.146:57458) 2024-01-21T20:03:10.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:10.259+0000 7f64c7fff700 1 -- 172.21.15.146:0/2804974352 learned_addr learned my addr 172.21.15.146:0/2804974352 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:03:10.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:10.260+0000 7f64c7fff700 1 -- 172.21.15.146:0/2804974352 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b8005040 con 0x7f64c8060280 2024-01-21T20:03:10.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:10.260+0000 7f64c7fff700 1 --2- 172.21.15.146:0/2804974352 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64c8060280 0x7f64c81066e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f64b8013510 tx=0x7f64b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:10.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:10.260+0000 7f64c4ff9700 1 -- 172.21.15.146:0/2804974352 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64b8009320 con 0x7f64c8060280 2024-01-21T20:03:10.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:10.260+0000 7f64ce3ab700 1 -- 172.21.15.146:0/2804974352 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64c8106dd0 con 0x7f64c8060280 2024-01-21T20:03:10.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:10.260+0000 7f64c4ff9700 1 -- 172.21.15.146:0/2804974352 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f64b8013f00 con 0x7f64c8060280 2024-01-21T20:03:10.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:10.260+0000 7f64c4ff9700 1 -- 172.21.15.146:0/2804974352 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64b8014dc0 con 0x7f64c8060280 2024-01-21T20:03:10.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:10.260+0000 7f64ce3ab700 1 -- 172.21.15.146:0/2804974352 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64c8107260 con 0x7f64c8060280 2024-01-21T20:03:10.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:10.261+0000 7f64c4ff9700 1 -- 172.21.15.146:0/2804974352 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f64b8014420 con 0x7f64c8060280 2024-01-21T20:03:10.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:10.261+0000 7f64ce3ab700 1 -- 172.21.15.146:0/2804974352 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64c8045ad0 con 0x7f64c8060280 2024-01-21T20:03:10.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:10.261+0000 7f64c4ff9700 1 --2- 172.21.15.146:0/2804974352 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f64b002ff60 0x7f64b0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:10.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:10.261+0000 7f64c4ff9700 1 -- 172.21.15.146:0/2804974352 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f64b8041110 con 0x7f64c8060280 2024-01-21T20:03:10.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:10.261+0000 7f64c77fe700 1 --2- 172.21.15.146:0/2804974352 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f64b002ff60 0x7f64b0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:10.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:10.262+0000 7f64c77fe700 1 --2- 172.21.15.146:0/2804974352 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f64b002ff60 0x7f64b0032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f64bc0096a0 tx=0x7f64bc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:10.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:10.266+0000 7f64c4ff9700 1 -- 172.21.15.146:0/2804974352 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f64b80095f0 con 0x7f64c8060280 2024-01-21T20:03:10.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:10.471+0000 7f64ce3ab700 1 -- 172.21.15.146:0/2804974352 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f64c8108110 con 0x7f64c8060280 2024-01-21T20:03:10.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:10.472+0000 7f64c4ff9700 1 -- 172.21.15.146:0/2804974352 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f64b801cb10 con 0x7f64c8060280 2024-01-21T20:03:10.463 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:03:10.463 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:03:10.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:10.473+0000 7f64ce3ab700 1 -- 172.21.15.146:0/2804974352 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f64b002ff60 msgr2=0x7f64b0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:10.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:10.473+0000 7f64ce3ab700 1 --2- 172.21.15.146:0/2804974352 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f64b002ff60 0x7f64b0032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f64bc0096a0 tx=0x7f64bc006b40).stop 2024-01-21T20:03:10.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:10.473+0000 7f64ce3ab700 1 -- 172.21.15.146:0/2804974352 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64c8060280 msgr2=0x7f64c81066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:10.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:10.473+0000 7f64ce3ab700 1 --2- 172.21.15.146:0/2804974352 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64c8060280 0x7f64c81066e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f64b8013510 tx=0x7f64b800b040).stop 2024-01-21T20:03:10.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:10.474+0000 7f64ce3ab700 1 -- 172.21.15.146:0/2804974352 shutdown_connections 2024-01-21T20:03:10.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:10.474+0000 7f64ce3ab700 1 --2- 172.21.15.146:0/2804974352 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f64b002ff60 0x7f64b0032420 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:10.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:10.474+0000 7f64ce3ab700 1 --2- 172.21.15.146:0/2804974352 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64c8060280 0x7f64c81066e0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:10.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:10.474+0000 7f64ce3ab700 1 -- 172.21.15.146:0/2804974352 >> 172.21.15.146:0/2804974352 conn(0x7f64c80f0b70 msgr2=0x7f64c80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:10.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:10.474+0000 7f64ce3ab700 1 -- 172.21.15.146:0/2804974352 shutdown_connections 2024-01-21T20:03:10.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:10.474+0000 7f64ce3ab700 1 -- 172.21.15.146:0/2804974352 wait complete. 2024-01-21T20:03:10.466 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:03:11.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:10 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:03:09.163146+0000 mgr.smithi053.fgsmol (mgr.14164) 121 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:11.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:10 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:03:10.473414+0000 mon.smithi053 (mon.0) 199 : audit [DBG] from='client.? 172.21.15.146:0/2804974352' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:03:11.857 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:03:11.858 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:03:13.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:13.041+0000 7f9da65f5700 1 -- 172.21.15.146:0/3098353077 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9da00f6510 msgr2=0x7f9da00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:13.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:13.041+0000 7f9da65f5700 1 --2- 172.21.15.146:0/3098353077 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9da00f6510 0x7f9da00f6930 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f9d94009fb0 tx=0x7f9d94009b30).stop 2024-01-21T20:03:13.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:13.041+0000 7f9da65f5700 1 -- 172.21.15.146:0/3098353077 shutdown_connections 2024-01-21T20:03:13.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:13.041+0000 7f9da65f5700 1 --2- 172.21.15.146:0/3098353077 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9da00f6510 0x7f9da00f6930 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:13.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:13.041+0000 7f9da65f5700 1 -- 172.21.15.146:0/3098353077 >> 172.21.15.146:0/3098353077 conn(0x7f9da00f1c70 msgr2=0x7f9da00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:13.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:13.041+0000 7f9da65f5700 1 -- 172.21.15.146:0/3098353077 shutdown_connections 2024-01-21T20:03:13.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:13.041+0000 7f9da65f5700 1 -- 172.21.15.146:0/3098353077 wait complete. 2024-01-21T20:03:13.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:13.042+0000 7f9da65f5700 1 Processor -- start 2024-01-21T20:03:13.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:13.042+0000 7f9da65f5700 1 -- start start 2024-01-21T20:03:13.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:13.042+0000 7f9da65f5700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9da00f6510 0x7f9da0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:13.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:13.042+0000 7f9da65f5700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da0106c20 con 0x7f9da00f6510 2024-01-21T20:03:13.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:13.042+0000 7f9d9ffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9da00f6510 0x7f9da0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:13.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:13.042+0000 7f9d9ffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9da00f6510 0x7f9da0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:57474/0 (socket says 172.21.15.146:57474) 2024-01-21T20:03:13.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:13.042+0000 7f9d9ffff700 1 -- 172.21.15.146:0/1999033342 learned_addr learned my addr 172.21.15.146:0/1999033342 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:03:13.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:13.043+0000 7f9d9ffff700 1 -- 172.21.15.146:0/1999033342 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d94005040 con 0x7f9da00f6510 2024-01-21T20:03:13.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:13.043+0000 7f9d9ffff700 1 --2- 172.21.15.146:0/1999033342 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9da00f6510 0x7f9da0106730 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f9d94013510 tx=0x7f9d9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:13.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:13.043+0000 7f9d9cff9700 1 -- 172.21.15.146:0/1999033342 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d94009320 con 0x7f9da00f6510 2024-01-21T20:03:13.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:13.043+0000 7f9da65f5700 1 -- 172.21.15.146:0/1999033342 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9da0106e20 con 0x7f9da00f6510 2024-01-21T20:03:13.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:13.043+0000 7f9d9cff9700 1 -- 172.21.15.146:0/1999033342 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9d94013f00 con 0x7f9da00f6510 2024-01-21T20:03:13.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:13.043+0000 7f9d9cff9700 1 -- 172.21.15.146:0/1999033342 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d94014d90 con 0x7f9da00f6510 2024-01-21T20:03:13.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:13.043+0000 7f9da65f5700 1 -- 172.21.15.146:0/1999033342 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9da01072b0 con 0x7f9da00f6510 2024-01-21T20:03:13.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:13.044+0000 7f9d9cff9700 1 -- 172.21.15.146:0/1999033342 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9d94014420 con 0x7f9da00f6510 2024-01-21T20:03:13.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:13.044+0000 7f9da65f5700 1 -- 172.21.15.146:0/1999033342 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9da0045ad0 con 0x7f9da00f6510 2024-01-21T20:03:13.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:13.044+0000 7f9d9cff9700 1 --2- 172.21.15.146:0/1999033342 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f9d8802ffb0 0x7f9d88032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:13.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:13.044+0000 7f9d9cff9700 1 -- 172.21.15.146:0/1999033342 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9d940410c0 con 0x7f9da00f6510 2024-01-21T20:03:13.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:13.044+0000 7f9d9f7fe700 1 --2- 172.21.15.146:0/1999033342 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f9d8802ffb0 0x7f9d88032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:13.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:13.045+0000 7f9d9f7fe700 1 --2- 172.21.15.146:0/1999033342 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f9d8802ffb0 0x7f9d88032470 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f9d900096a0 tx=0x7f9d90006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:13.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:13.049+0000 7f9d9cff9700 1 -- 172.21.15.146:0/1999033342 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9d940095f0 con 0x7f9da00f6510 2024-01-21T20:03:13.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:12 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:03:11.163516+0000 mgr.smithi053.fgsmol (mgr.14164) 122 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:13.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:13.253+0000 7f9da65f5700 1 -- 172.21.15.146:0/1999033342 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9da00fbf60 con 0x7f9da00f6510 2024-01-21T20:03:13.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:13.254+0000 7f9d9cff9700 1 -- 172.21.15.146:0/1999033342 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9d9401db70 con 0x7f9da00f6510 2024-01-21T20:03:13.245 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:03:13.245 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:03:13.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:13.255+0000 7f9da65f5700 1 -- 172.21.15.146:0/1999033342 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f9d8802ffb0 msgr2=0x7f9d88032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:13.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:13.255+0000 7f9da65f5700 1 --2- 172.21.15.146:0/1999033342 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f9d8802ffb0 0x7f9d88032470 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f9d900096a0 tx=0x7f9d90006b40).stop 2024-01-21T20:03:13.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:13.255+0000 7f9da65f5700 1 -- 172.21.15.146:0/1999033342 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9da00f6510 msgr2=0x7f9da0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:13.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:13.255+0000 7f9da65f5700 1 --2- 172.21.15.146:0/1999033342 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9da00f6510 0x7f9da0106730 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f9d94013510 tx=0x7f9d9400b040).stop 2024-01-21T20:03:13.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:13.256+0000 7f9da65f5700 1 -- 172.21.15.146:0/1999033342 shutdown_connections 2024-01-21T20:03:13.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:13.256+0000 7f9da65f5700 1 --2- 172.21.15.146:0/1999033342 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f9d8802ffb0 0x7f9d88032470 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:13.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:13.256+0000 7f9da65f5700 1 --2- 172.21.15.146:0/1999033342 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9da00f6510 0x7f9da0106730 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:13.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:13.256+0000 7f9da65f5700 1 -- 172.21.15.146:0/1999033342 >> 172.21.15.146:0/1999033342 conn(0x7f9da00f1c70 msgr2=0x7f9da00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:13.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:13.256+0000 7f9da65f5700 1 -- 172.21.15.146:0/1999033342 shutdown_connections 2024-01-21T20:03:13.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:13.256+0000 7f9da65f5700 1 -- 172.21.15.146:0/1999033342 wait complete. 2024-01-21T20:03:13.248 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:03:14.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:13 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:03:13.255462+0000 mon.smithi053 (mon.0) 200 : audit [DBG] from='client.? 172.21.15.146:0/1999033342' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:03:14.672 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:03:14.673 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:03:15.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:14 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:03:13.163890+0000 mgr.smithi053.fgsmol (mgr.14164) 123 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:15.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:15.918+0000 7f36a7a8b700 1 -- 172.21.15.146:0/3749912793 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36a00f7ae0 msgr2=0x7f36a00f7f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:15.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:15.918+0000 7f36a7a8b700 1 --2- 172.21.15.146:0/3749912793 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36a00f7ae0 0x7f36a00f7f00 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f369c009fb0 tx=0x7f369c009b30).stop 2024-01-21T20:03:15.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:15.918+0000 7f36a7a8b700 1 -- 172.21.15.146:0/3749912793 shutdown_connections 2024-01-21T20:03:15.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:15.918+0000 7f36a7a8b700 1 --2- 172.21.15.146:0/3749912793 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36a00f7ae0 0x7f36a00f7f00 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:15.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:15.918+0000 7f36a7a8b700 1 -- 172.21.15.146:0/3749912793 >> 172.21.15.146:0/3749912793 conn(0x7f36a00f0260 msgr2=0x7f36a00f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:15.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:15.918+0000 7f36a7a8b700 1 -- 172.21.15.146:0/3749912793 shutdown_connections 2024-01-21T20:03:15.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:15.918+0000 7f36a7a8b700 1 -- 172.21.15.146:0/3749912793 wait complete. 2024-01-21T20:03:15.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:15.919+0000 7f36a7a8b700 1 Processor -- start 2024-01-21T20:03:15.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:15.919+0000 7f36a7a8b700 1 -- start start 2024-01-21T20:03:15.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:15.919+0000 7f36a7a8b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36a00f7ae0 0x7f36a00f73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:15.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:15.919+0000 7f36a7a8b700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36a00f5830 con 0x7f36a00f7ae0 2024-01-21T20:03:15.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:15.919+0000 7f36a5827700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36a00f7ae0 0x7f36a00f73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:15.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:15.919+0000 7f36a5827700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36a00f7ae0 0x7f36a00f73b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:57490/0 (socket says 172.21.15.146:57490) 2024-01-21T20:03:15.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:15.919+0000 7f36a5827700 1 -- 172.21.15.146:0/3965078155 learned_addr learned my addr 172.21.15.146:0/3965078155 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:03:15.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:15.919+0000 7f36a5827700 1 -- 172.21.15.146:0/3965078155 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f369c005040 con 0x7f36a00f7ae0 2024-01-21T20:03:15.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:15.919+0000 7f36a5827700 1 --2- 172.21.15.146:0/3965078155 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36a00f7ae0 0x7f36a00f73b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f369c013510 tx=0x7f369c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:15.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:15.920+0000 7f36967fc700 1 -- 172.21.15.146:0/3965078155 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f369c009320 con 0x7f36a00f7ae0 2024-01-21T20:03:15.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:15.920+0000 7f36967fc700 1 -- 172.21.15.146:0/3965078155 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f369c013f00 con 0x7f36a00f7ae0 2024-01-21T20:03:15.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:15.920+0000 7f36a7a8b700 1 -- 172.21.15.146:0/3965078155 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36a00f59d0 con 0x7f36a00f7ae0 2024-01-21T20:03:15.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:15.920+0000 7f36967fc700 1 -- 172.21.15.146:0/3965078155 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f369c014d90 con 0x7f36a00f7ae0 2024-01-21T20:03:15.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:15.920+0000 7f36a7a8b700 1 -- 172.21.15.146:0/3965078155 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36a00f5e60 con 0x7f36a00f7ae0 2024-01-21T20:03:15.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:15.920+0000 7f36967fc700 1 -- 172.21.15.146:0/3965078155 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f369c0095f0 con 0x7f36a00f7ae0 2024-01-21T20:03:15.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:15.920+0000 7f36a7a8b700 1 -- 172.21.15.146:0/3965078155 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36a0059100 con 0x7f36a00f7ae0 2024-01-21T20:03:15.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:15.921+0000 7f36967fc700 1 --2- 172.21.15.146:0/3965078155 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f368c02ffb0 0x7f368c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:15.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:15.921+0000 7f36967fc700 1 -- 172.21.15.146:0/3965078155 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f369c040f90 con 0x7f36a00f7ae0 2024-01-21T20:03:15.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:15.921+0000 7f36a5026700 1 --2- 172.21.15.146:0/3965078155 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f368c02ffb0 0x7f368c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:15.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:15.921+0000 7f36a5026700 1 --2- 172.21.15.146:0/3965078155 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f368c02ffb0 0x7f368c032470 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f36900096a0 tx=0x7f3690006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:15.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:15.925+0000 7f36967fc700 1 -- 172.21.15.146:0/3965078155 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f369c01d7f0 con 0x7f36a00f7ae0 2024-01-21T20:03:16.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:16.130+0000 7f36a7a8b700 1 -- 172.21.15.146:0/3965078155 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f36a0045ad0 con 0x7f36a00f7ae0 2024-01-21T20:03:16.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:16.131+0000 7f36967fc700 1 -- 172.21.15.146:0/3965078155 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f369c010230 con 0x7f36a00f7ae0 2024-01-21T20:03:16.122 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:03:16.122 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:03:16.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:16.132+0000 7f36a7a8b700 1 -- 172.21.15.146:0/3965078155 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f368c02ffb0 msgr2=0x7f368c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:16.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:16.132+0000 7f36a7a8b700 1 --2- 172.21.15.146:0/3965078155 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f368c02ffb0 0x7f368c032470 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f36900096a0 tx=0x7f3690006b40).stop 2024-01-21T20:03:16.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:16.132+0000 7f36a7a8b700 1 -- 172.21.15.146:0/3965078155 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36a00f7ae0 msgr2=0x7f36a00f73b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:16.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:16.132+0000 7f36a7a8b700 1 --2- 172.21.15.146:0/3965078155 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36a00f7ae0 0x7f36a00f73b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f369c013510 tx=0x7f369c00b040).stop 2024-01-21T20:03:16.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:16.133+0000 7f36a7a8b700 1 -- 172.21.15.146:0/3965078155 shutdown_connections 2024-01-21T20:03:16.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:16.133+0000 7f36a7a8b700 1 --2- 172.21.15.146:0/3965078155 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f368c02ffb0 0x7f368c032470 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:16.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:16.133+0000 7f36a7a8b700 1 --2- 172.21.15.146:0/3965078155 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36a00f7ae0 0x7f36a00f73b0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:16.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:16.133+0000 7f36a7a8b700 1 -- 172.21.15.146:0/3965078155 >> 172.21.15.146:0/3965078155 conn(0x7f36a00f0260 msgr2=0x7f36a00f0d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:16.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:16.133+0000 7f36a7a8b700 1 -- 172.21.15.146:0/3965078155 shutdown_connections 2024-01-21T20:03:16.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:16.133+0000 7f36a7a8b700 1 -- 172.21.15.146:0/3965078155 wait complete. 2024-01-21T20:03:16.125 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:03:17.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:03:15.164207+0000 mgr.smithi053.fgsmol (mgr.14164) 124 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:17.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:03:16.132312+0000 mon.smithi053 (mon.0) 201 : audit [DBG] from='client.? 172.21.15.146:0/3965078155' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:03:17.565 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:03:17.565 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:03:18.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:18.750+0000 7fcbc359e700 1 -- 172.21.15.146:0/3572384398 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcbc40f51a0 msgr2=0x7fcbc40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:18.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:18.750+0000 7fcbc359e700 1 --2- 172.21.15.146:0/3572384398 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcbc40f51a0 0x7fcbc40f55c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7fcbb8009fb0 tx=0x7fcbb8009b30).stop 2024-01-21T20:03:18.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:18.751+0000 7fcbc359e700 1 -- 172.21.15.146:0/3572384398 shutdown_connections 2024-01-21T20:03:18.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:18.751+0000 7fcbc359e700 1 --2- 172.21.15.146:0/3572384398 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcbc40f51a0 0x7fcbc40f55c0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:18.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:18.751+0000 7fcbc359e700 1 -- 172.21.15.146:0/3572384398 >> 172.21.15.146:0/3572384398 conn(0x7fcbc40f09a0 msgr2=0x7fcbc40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:18.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:18.751+0000 7fcbc359e700 1 -- 172.21.15.146:0/3572384398 shutdown_connections 2024-01-21T20:03:18.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:18.751+0000 7fcbc359e700 1 -- 172.21.15.146:0/3572384398 wait complete. 2024-01-21T20:03:18.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:18.751+0000 7fcbc359e700 1 Processor -- start 2024-01-21T20:03:18.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:18.751+0000 7fcbc359e700 1 -- start start 2024-01-21T20:03:18.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:18.752+0000 7fcbc359e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcbc40f51a0 0x7fcbc41053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:18.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:18.752+0000 7fcbc359e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc41058a0 con 0x7fcbc40f51a0 2024-01-21T20:03:18.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:18.752+0000 7fcbc259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcbc40f51a0 0x7fcbc41053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:18.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:18.752+0000 7fcbc259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcbc40f51a0 0x7fcbc41053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:59378/0 (socket says 172.21.15.146:59378) 2024-01-21T20:03:18.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:18.752+0000 7fcbc259c700 1 -- 172.21.15.146:0/1168422600 learned_addr learned my addr 172.21.15.146:0/1168422600 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:03:18.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:18.752+0000 7fcbc259c700 1 -- 172.21.15.146:0/1168422600 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbb8005040 con 0x7fcbc40f51a0 2024-01-21T20:03:18.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:18.752+0000 7fcbc259c700 1 --2- 172.21.15.146:0/1168422600 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcbc40f51a0 0x7fcbc41053b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fcbb8013510 tx=0x7fcbb800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:18.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:18.752+0000 7fcbb2ffd700 1 -- 172.21.15.146:0/1168422600 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcbb8009320 con 0x7fcbc40f51a0 2024-01-21T20:03:18.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:18.752+0000 7fcbc359e700 1 -- 172.21.15.146:0/1168422600 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbc4105aa0 con 0x7fcbc40f51a0 2024-01-21T20:03:18.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:18.752+0000 7fcbb2ffd700 1 -- 172.21.15.146:0/1168422600 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcbb8013f00 con 0x7fcbc40f51a0 2024-01-21T20:03:18.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:18.752+0000 7fcbb2ffd700 1 -- 172.21.15.146:0/1168422600 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcbb8014dc0 con 0x7fcbc40f51a0 2024-01-21T20:03:18.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:18.752+0000 7fcbc359e700 1 -- 172.21.15.146:0/1168422600 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbc4105f30 con 0x7fcbc40f51a0 2024-01-21T20:03:18.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:18.753+0000 7fcbb2ffd700 1 -- 172.21.15.146:0/1168422600 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcbb80095f0 con 0x7fcbc40f51a0 2024-01-21T20:03:18.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:18.753+0000 7fcbc359e700 1 -- 172.21.15.146:0/1168422600 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbc4045ad0 con 0x7fcbc40f51a0 2024-01-21T20:03:18.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:18.753+0000 7fcbb2ffd700 1 --2- 172.21.15.146:0/1168422600 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fcbac02ff60 0x7fcbac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:18.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:18.753+0000 7fcbb2ffd700 1 -- 172.21.15.146:0/1168422600 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcbb8041040 con 0x7fcbc40f51a0 2024-01-21T20:03:18.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:18.753+0000 7fcbc1d9b700 1 --2- 172.21.15.146:0/1168422600 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fcbac02ff60 0x7fcbac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:18.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:18.754+0000 7fcbc1d9b700 1 --2- 172.21.15.146:0/1168422600 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fcbac02ff60 0x7fcbac032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fcbb40096a0 tx=0x7fcbb4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:18.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:18.758+0000 7fcbb2ffd700 1 -- 172.21.15.146:0/1168422600 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcbb8014420 con 0x7fcbc40f51a0 2024-01-21T20:03:18.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:18.965+0000 7fcbc359e700 1 -- 172.21.15.146:0/1168422600 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcbc4106de0 con 0x7fcbc40f51a0 2024-01-21T20:03:18.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:18.966+0000 7fcbb2ffd700 1 -- 172.21.15.146:0/1168422600 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcbb8010230 con 0x7fcbc40f51a0 2024-01-21T20:03:18.957 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:03:18.957 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:03:18.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:18.967+0000 7fcbc359e700 1 -- 172.21.15.146:0/1168422600 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fcbac02ff60 msgr2=0x7fcbac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:18.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:18.967+0000 7fcbc359e700 1 --2- 172.21.15.146:0/1168422600 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fcbac02ff60 0x7fcbac032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fcbb40096a0 tx=0x7fcbb4006b40).stop 2024-01-21T20:03:18.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:18.967+0000 7fcbc359e700 1 -- 172.21.15.146:0/1168422600 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcbc40f51a0 msgr2=0x7fcbc41053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:18.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:18.967+0000 7fcbc359e700 1 --2- 172.21.15.146:0/1168422600 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcbc40f51a0 0x7fcbc41053b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fcbb8013510 tx=0x7fcbb800b040).stop 2024-01-21T20:03:18.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:18.967+0000 7fcbc359e700 1 -- 172.21.15.146:0/1168422600 shutdown_connections 2024-01-21T20:03:18.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:18.967+0000 7fcbc359e700 1 --2- 172.21.15.146:0/1168422600 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fcbac02ff60 0x7fcbac032420 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:18.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:18.967+0000 7fcbc359e700 1 --2- 172.21.15.146:0/1168422600 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcbc40f51a0 0x7fcbc41053b0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:18.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:18.967+0000 7fcbc359e700 1 -- 172.21.15.146:0/1168422600 >> 172.21.15.146:0/1168422600 conn(0x7fcbc40f09a0 msgr2=0x7fcbc40f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:18.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:18.967+0000 7fcbc359e700 1 -- 172.21.15.146:0/1168422600 shutdown_connections 2024-01-21T20:03:18.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:18.967+0000 7fcbc359e700 1 -- 172.21.15.146:0/1168422600 wait complete. 2024-01-21T20:03:18.959 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:03:19.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:18 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:03:17.164525+0000 mgr.smithi053.fgsmol (mgr.14164) 125 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:20.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:19 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:03:18.966897+0000 mon.smithi053 (mon.0) 202 : audit [DBG] from='client.? 172.21.15.146:0/1168422600' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:03:20.398 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:03:20.398 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:03:21.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:20 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:03:19.164851+0000 mgr.smithi053.fgsmol (mgr.14164) 126 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:21.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:21.628+0000 7fdd245d2700 1 -- 172.21.15.146:0/3848283052 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd1c060280 msgr2=0x7fdd1c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:21.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:21.628+0000 7fdd245d2700 1 --2- 172.21.15.146:0/3848283052 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd1c060280 0x7fdd1c0f90f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7fdd0c009fb0 tx=0x7fdd0c009b30).stop 2024-01-21T20:03:21.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:21.628+0000 7fdd245d2700 1 -- 172.21.15.146:0/3848283052 shutdown_connections 2024-01-21T20:03:21.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:21.628+0000 7fdd245d2700 1 --2- 172.21.15.146:0/3848283052 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd1c060280 0x7fdd1c0f90f0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:21.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:21.628+0000 7fdd245d2700 1 -- 172.21.15.146:0/3848283052 >> 172.21.15.146:0/3848283052 conn(0x7fdd1c0f0b70 msgr2=0x7fdd1c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:21.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:21.628+0000 7fdd245d2700 1 -- 172.21.15.146:0/3848283052 shutdown_connections 2024-01-21T20:03:21.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:21.628+0000 7fdd245d2700 1 -- 172.21.15.146:0/3848283052 wait complete. 2024-01-21T20:03:21.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:21.629+0000 7fdd245d2700 1 Processor -- start 2024-01-21T20:03:21.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:21.629+0000 7fdd245d2700 1 -- start start 2024-01-21T20:03:21.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:21.629+0000 7fdd245d2700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd1c060280 0x7fdd1c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:21.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:21.629+0000 7fdd245d2700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1c106bd0 con 0x7fdd1c060280 2024-01-21T20:03:21.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:21.629+0000 7fdd2236e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd1c060280 0x7fdd1c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:21.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:21.629+0000 7fdd2236e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd1c060280 0x7fdd1c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:59390/0 (socket says 172.21.15.146:59390) 2024-01-21T20:03:21.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:21.629+0000 7fdd2236e700 1 -- 172.21.15.146:0/771597815 learned_addr learned my addr 172.21.15.146:0/771597815 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:03:21.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:21.629+0000 7fdd2236e700 1 -- 172.21.15.146:0/771597815 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd0c005040 con 0x7fdd1c060280 2024-01-21T20:03:21.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:21.629+0000 7fdd2236e700 1 --2- 172.21.15.146:0/771597815 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd1c060280 0x7fdd1c1066e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fdd0c013510 tx=0x7fdd0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:21.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:21.630+0000 7fdd12ffd700 1 -- 172.21.15.146:0/771597815 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd0c009320 con 0x7fdd1c060280 2024-01-21T20:03:21.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:21.630+0000 7fdd245d2700 1 -- 172.21.15.146:0/771597815 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd1c106dd0 con 0x7fdd1c060280 2024-01-21T20:03:21.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:21.630+0000 7fdd12ffd700 1 -- 172.21.15.146:0/771597815 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdd0c013f00 con 0x7fdd1c060280 2024-01-21T20:03:21.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:21.630+0000 7fdd245d2700 1 -- 172.21.15.146:0/771597815 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd1c107260 con 0x7fdd1c060280 2024-01-21T20:03:21.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:21.630+0000 7fdd12ffd700 1 -- 172.21.15.146:0/771597815 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd0c014dc0 con 0x7fdd1c060280 2024-01-21T20:03:21.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:21.630+0000 7fdd245d2700 1 -- 172.21.15.146:0/771597815 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd1c045ad0 con 0x7fdd1c060280 2024-01-21T20:03:21.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:21.630+0000 7fdd12ffd700 1 -- 172.21.15.146:0/771597815 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdd0c0095f0 con 0x7fdd1c060280 2024-01-21T20:03:21.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:21.631+0000 7fdd12ffd700 1 --2- 172.21.15.146:0/771597815 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fdd0802ff60 0x7fdd08032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:21.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:21.631+0000 7fdd12ffd700 1 -- 172.21.15.146:0/771597815 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdd0c0410b0 con 0x7fdd1c060280 2024-01-21T20:03:21.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:21.631+0000 7fdd21b6d700 1 --2- 172.21.15.146:0/771597815 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fdd0802ff60 0x7fdd08032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:21.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:21.631+0000 7fdd21b6d700 1 --2- 172.21.15.146:0/771597815 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fdd0802ff60 0x7fdd08032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fdd180096a0 tx=0x7fdd18006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:21.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:21.635+0000 7fdd12ffd700 1 -- 172.21.15.146:0/771597815 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdd0c014420 con 0x7fdd1c060280 2024-01-21T20:03:21.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:21.838+0000 7fdd245d2700 1 -- 172.21.15.146:0/771597815 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdd1c1080b0 con 0x7fdd1c060280 2024-01-21T20:03:21.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:21.839+0000 7fdd12ffd700 1 -- 172.21.15.146:0/771597815 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdd0c010230 con 0x7fdd1c060280 2024-01-21T20:03:21.830 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:03:21.830 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:03:21.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:21.840+0000 7fdd245d2700 1 -- 172.21.15.146:0/771597815 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fdd0802ff60 msgr2=0x7fdd08032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:21.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:21.840+0000 7fdd245d2700 1 --2- 172.21.15.146:0/771597815 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fdd0802ff60 0x7fdd08032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fdd180096a0 tx=0x7fdd18006b40).stop 2024-01-21T20:03:21.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:21.840+0000 7fdd245d2700 1 -- 172.21.15.146:0/771597815 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd1c060280 msgr2=0x7fdd1c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:21.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:21.840+0000 7fdd245d2700 1 --2- 172.21.15.146:0/771597815 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd1c060280 0x7fdd1c1066e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fdd0c013510 tx=0x7fdd0c00b040).stop 2024-01-21T20:03:21.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:21.840+0000 7fdd245d2700 1 -- 172.21.15.146:0/771597815 shutdown_connections 2024-01-21T20:03:21.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:21.840+0000 7fdd245d2700 1 --2- 172.21.15.146:0/771597815 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fdd0802ff60 0x7fdd08032420 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:21.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:21.840+0000 7fdd245d2700 1 --2- 172.21.15.146:0/771597815 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd1c060280 0x7fdd1c1066e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:21.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:21.840+0000 7fdd245d2700 1 -- 172.21.15.146:0/771597815 >> 172.21.15.146:0/771597815 conn(0x7fdd1c0f0b70 msgr2=0x7fdd1c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:21.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:21.840+0000 7fdd245d2700 1 -- 172.21.15.146:0/771597815 shutdown_connections 2024-01-21T20:03:21.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:21.840+0000 7fdd245d2700 1 -- 172.21.15.146:0/771597815 wait complete. 2024-01-21T20:03:21.832 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:03:23.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:03:21.165296+0000 mgr.smithi053.fgsmol (mgr.14164) 127 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:23.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:03:21.840320+0000 mon.smithi053 (mon.0) 203 : audit [DBG] from='client.? 172.21.15.146:0/771597815' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:03:23.242 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:03:23.242 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:03:24.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:24.475+0000 7fb106897700 1 -- 172.21.15.146:0/938611962 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1000f5410 msgr2=0x7fb1000f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:24.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:24.475+0000 7fb106897700 1 --2- 172.21.15.146:0/938611962 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1000f5410 0x7fb1000f5830 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7fb0f0009fb0 tx=0x7fb0f0009b30).stop 2024-01-21T20:03:24.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:24.475+0000 7fb106897700 1 -- 172.21.15.146:0/938611962 shutdown_connections 2024-01-21T20:03:24.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:24.475+0000 7fb106897700 1 --2- 172.21.15.146:0/938611962 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1000f5410 0x7fb1000f5830 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:24.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:24.475+0000 7fb106897700 1 -- 172.21.15.146:0/938611962 >> 172.21.15.146:0/938611962 conn(0x7fb1000f0b70 msgr2=0x7fb1000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:24.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:24.475+0000 7fb106897700 1 -- 172.21.15.146:0/938611962 shutdown_connections 2024-01-21T20:03:24.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:24.475+0000 7fb106897700 1 -- 172.21.15.146:0/938611962 wait complete. 2024-01-21T20:03:24.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:24.476+0000 7fb106897700 1 Processor -- start 2024-01-21T20:03:24.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:24.476+0000 7fb106897700 1 -- start start 2024-01-21T20:03:24.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:24.476+0000 7fb106897700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1000f5410 0x7fb1000fb580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:24.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:24.476+0000 7fb106897700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1000f9a00 con 0x7fb1000f5410 2024-01-21T20:03:24.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:24.476+0000 7fb0fffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1000f5410 0x7fb1000fb580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:24.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:24.476+0000 7fb0fffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1000f5410 0x7fb1000fb580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:59406/0 (socket says 172.21.15.146:59406) 2024-01-21T20:03:24.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:24.476+0000 7fb0fffff700 1 -- 172.21.15.146:0/3117956513 learned_addr learned my addr 172.21.15.146:0/3117956513 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:03:24.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:24.477+0000 7fb0fffff700 1 -- 172.21.15.146:0/3117956513 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0f0005040 con 0x7fb1000f5410 2024-01-21T20:03:24.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:24.477+0000 7fb0fffff700 1 --2- 172.21.15.146:0/3117956513 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1000f5410 0x7fb1000fb580 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fb0f0013510 tx=0x7fb0f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:24.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:24.477+0000 7fb0fcff9700 1 -- 172.21.15.146:0/3117956513 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0f0009320 con 0x7fb1000f5410 2024-01-21T20:03:24.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:24.477+0000 7fb106897700 1 -- 172.21.15.146:0/3117956513 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1000f9ba0 con 0x7fb1000f5410 2024-01-21T20:03:24.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:24.477+0000 7fb0fcff9700 1 -- 172.21.15.146:0/3117956513 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb0f0013f00 con 0x7fb1000f5410 2024-01-21T20:03:24.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:24.477+0000 7fb0fcff9700 1 -- 172.21.15.146:0/3117956513 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0f0014d90 con 0x7fb1000f5410 2024-01-21T20:03:24.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:24.477+0000 7fb106897700 1 -- 172.21.15.146:0/3117956513 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1000fa030 con 0x7fb1000f5410 2024-01-21T20:03:24.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:24.478+0000 7fb0fcff9700 1 -- 172.21.15.146:0/3117956513 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb0f00095f0 con 0x7fb1000f5410 2024-01-21T20:03:24.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:24.478+0000 7fb106897700 1 -- 172.21.15.146:0/3117956513 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb100059100 con 0x7fb1000f5410 2024-01-21T20:03:24.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:24.478+0000 7fb0fcff9700 1 --2- 172.21.15.146:0/3117956513 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb0e802ff60 0x7fb0e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:24.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:24.478+0000 7fb0fcff9700 1 -- 172.21.15.146:0/3117956513 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb0f00410a0 con 0x7fb1000f5410 2024-01-21T20:03:24.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:24.478+0000 7fb0ff7fe700 1 --2- 172.21.15.146:0/3117956513 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb0e802ff60 0x7fb0e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:24.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:24.479+0000 7fb0ff7fe700 1 --2- 172.21.15.146:0/3117956513 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb0e802ff60 0x7fb0e8032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fb0f40096a0 tx=0x7fb0f4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:24.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:24.483+0000 7fb0fcff9700 1 -- 172.21.15.146:0/3117956513 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb0f001d7f0 con 0x7fb1000f5410 2024-01-21T20:03:24.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:24.686+0000 7fb106897700 1 -- 172.21.15.146:0/3117956513 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb1000fbf90 con 0x7fb1000f5410 2024-01-21T20:03:24.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:24.687+0000 7fb0fcff9700 1 -- 172.21.15.146:0/3117956513 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb0f0010230 con 0x7fb1000f5410 2024-01-21T20:03:24.678 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:03:24.678 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:03:24.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:24.689+0000 7fb106897700 1 -- 172.21.15.146:0/3117956513 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb0e802ff60 msgr2=0x7fb0e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:24.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:24.689+0000 7fb106897700 1 --2- 172.21.15.146:0/3117956513 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb0e802ff60 0x7fb0e8032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fb0f40096a0 tx=0x7fb0f4006b40).stop 2024-01-21T20:03:24.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:24.689+0000 7fb106897700 1 -- 172.21.15.146:0/3117956513 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1000f5410 msgr2=0x7fb1000fb580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:24.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:24.689+0000 7fb106897700 1 --2- 172.21.15.146:0/3117956513 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1000f5410 0x7fb1000fb580 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fb0f0013510 tx=0x7fb0f000b040).stop 2024-01-21T20:03:24.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:24.689+0000 7fb106897700 1 -- 172.21.15.146:0/3117956513 shutdown_connections 2024-01-21T20:03:24.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:24.689+0000 7fb106897700 1 --2- 172.21.15.146:0/3117956513 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb0e802ff60 0x7fb0e8032420 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:24.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:24.689+0000 7fb106897700 1 --2- 172.21.15.146:0/3117956513 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1000f5410 0x7fb1000fb580 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:24.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:24.689+0000 7fb106897700 1 -- 172.21.15.146:0/3117956513 >> 172.21.15.146:0/3117956513 conn(0x7fb1000f0b70 msgr2=0x7fb1000f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:24.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:24.689+0000 7fb106897700 1 -- 172.21.15.146:0/3117956513 shutdown_connections 2024-01-21T20:03:24.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:24.689+0000 7fb106897700 1 -- 172.21.15.146:0/3117956513 wait complete. 2024-01-21T20:03:24.681 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:03:25.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:24 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:03:23.165662+0000 mgr.smithi053.fgsmol (mgr.14164) 128 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:25.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:24 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:03:24.688561+0000 mon.smithi053 (mon.0) 204 : audit [DBG] from='client.? 172.21.15.146:0/3117956513' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:03:26.093 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:03:26.094 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:03:27.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:26 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:03:25.165913+0000 mgr.smithi053.fgsmol (mgr.14164) 129 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:27.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:27.324+0000 7f4b549b9700 1 -- 172.21.15.146:0/229065659 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b500601e0 msgr2=0x7f4b500f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:27.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:27.324+0000 7f4b549b9700 1 --2- 172.21.15.146:0/229065659 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b500601e0 0x7f4b500f8f20 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f4b38009fb0 tx=0x7f4b38009b30).stop 2024-01-21T20:03:27.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:27.324+0000 7f4b549b9700 1 -- 172.21.15.146:0/229065659 shutdown_connections 2024-01-21T20:03:27.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:27.324+0000 7f4b549b9700 1 --2- 172.21.15.146:0/229065659 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b500601e0 0x7f4b500f8f20 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:27.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:27.324+0000 7f4b549b9700 1 -- 172.21.15.146:0/229065659 >> 172.21.15.146:0/229065659 conn(0x7f4b500f09a0 msgr2=0x7f4b500f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:27.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:27.324+0000 7f4b549b9700 1 -- 172.21.15.146:0/229065659 shutdown_connections 2024-01-21T20:03:27.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:27.324+0000 7f4b549b9700 1 -- 172.21.15.146:0/229065659 wait complete. 2024-01-21T20:03:27.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:27.324+0000 7f4b549b9700 1 Processor -- start 2024-01-21T20:03:27.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:27.325+0000 7f4b549b9700 1 -- start start 2024-01-21T20:03:27.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:27.325+0000 7f4b549b9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b500601e0 0x7f4b50106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:27.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:27.325+0000 7f4b549b9700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b501069f0 con 0x7f4b500601e0 2024-01-21T20:03:27.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:27.325+0000 7f4b4f7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b500601e0 0x7f4b50106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:27.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:27.325+0000 7f4b4f7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b500601e0 0x7f4b50106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:34036/0 (socket says 172.21.15.146:34036) 2024-01-21T20:03:27.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:27.325+0000 7f4b4f7fe700 1 -- 172.21.15.146:0/1205386812 learned_addr learned my addr 172.21.15.146:0/1205386812 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:03:27.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:27.325+0000 7f4b4f7fe700 1 -- 172.21.15.146:0/1205386812 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b38005040 con 0x7f4b500601e0 2024-01-21T20:03:27.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:27.325+0000 7f4b4f7fe700 1 --2- 172.21.15.146:0/1205386812 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b500601e0 0x7f4b50106500 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f4b38013510 tx=0x7f4b3800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:27.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:27.326+0000 7f4b47fff700 1 -- 172.21.15.146:0/1205386812 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b38009320 con 0x7f4b500601e0 2024-01-21T20:03:27.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:27.326+0000 7f4b549b9700 1 -- 172.21.15.146:0/1205386812 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b50106bf0 con 0x7f4b500601e0 2024-01-21T20:03:27.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:27.326+0000 7f4b47fff700 1 -- 172.21.15.146:0/1205386812 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4b38013f00 con 0x7f4b500601e0 2024-01-21T20:03:27.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:27.326+0000 7f4b47fff700 1 -- 172.21.15.146:0/1205386812 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b38014d90 con 0x7f4b500601e0 2024-01-21T20:03:27.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:27.326+0000 7f4b549b9700 1 -- 172.21.15.146:0/1205386812 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b50107080 con 0x7f4b500601e0 2024-01-21T20:03:27.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:27.327+0000 7f4b47fff700 1 -- 172.21.15.146:0/1205386812 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4b380095f0 con 0x7f4b500601e0 2024-01-21T20:03:27.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:27.327+0000 7f4b549b9700 1 -- 172.21.15.146:0/1205386812 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b50045ad0 con 0x7f4b500601e0 2024-01-21T20:03:27.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:27.327+0000 7f4b47fff700 1 --2- 172.21.15.146:0/1205386812 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f4b3c02ff60 0x7f4b3c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:27.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:27.327+0000 7f4b47fff700 1 -- 172.21.15.146:0/1205386812 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4b38041080 con 0x7f4b500601e0 2024-01-21T20:03:27.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:27.327+0000 7f4b4effd700 1 --2- 172.21.15.146:0/1205386812 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f4b3c02ff60 0x7f4b3c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:27.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:27.327+0000 7f4b4effd700 1 --2- 172.21.15.146:0/1205386812 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f4b3c02ff60 0x7f4b3c032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f4b400096a0 tx=0x7f4b40006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:27.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:27.331+0000 7f4b47fff700 1 -- 172.21.15.146:0/1205386812 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4b38014420 con 0x7f4b500601e0 2024-01-21T20:03:27.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:27.534+0000 7f4b549b9700 1 -- 172.21.15.146:0/1205386812 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4b50107ed0 con 0x7f4b500601e0 2024-01-21T20:03:27.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:27.535+0000 7f4b47fff700 1 -- 172.21.15.146:0/1205386812 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4b38010230 con 0x7f4b500601e0 2024-01-21T20:03:27.526 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:03:27.526 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:03:27.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:27.537+0000 7f4b549b9700 1 -- 172.21.15.146:0/1205386812 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f4b3c02ff60 msgr2=0x7f4b3c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:27.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:27.537+0000 7f4b549b9700 1 --2- 172.21.15.146:0/1205386812 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f4b3c02ff60 0x7f4b3c032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f4b400096a0 tx=0x7f4b40006b40).stop 2024-01-21T20:03:27.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:27.537+0000 7f4b549b9700 1 -- 172.21.15.146:0/1205386812 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b500601e0 msgr2=0x7f4b50106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:27.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:27.537+0000 7f4b549b9700 1 --2- 172.21.15.146:0/1205386812 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b500601e0 0x7f4b50106500 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f4b38013510 tx=0x7f4b3800b040).stop 2024-01-21T20:03:27.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:27.537+0000 7f4b549b9700 1 -- 172.21.15.146:0/1205386812 shutdown_connections 2024-01-21T20:03:27.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:27.537+0000 7f4b549b9700 1 --2- 172.21.15.146:0/1205386812 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f4b3c02ff60 0x7f4b3c032420 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:27.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:27.537+0000 7f4b549b9700 1 --2- 172.21.15.146:0/1205386812 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b500601e0 0x7f4b50106500 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:27.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:27.537+0000 7f4b549b9700 1 -- 172.21.15.146:0/1205386812 >> 172.21.15.146:0/1205386812 conn(0x7f4b500f09a0 msgr2=0x7f4b500f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:27.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:27.537+0000 7f4b549b9700 1 -- 172.21.15.146:0/1205386812 shutdown_connections 2024-01-21T20:03:27.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:27.537+0000 7f4b549b9700 1 -- 172.21.15.146:0/1205386812 wait complete. 2024-01-21T20:03:27.529 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:03:28.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:27 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:03:27.536357+0000 mon.smithi053 (mon.0) 205 : audit [DBG] from='client.? 172.21.15.146:0/1205386812' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:03:28.942 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:03:28.942 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:03:29.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:28 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:03:27.166258+0000 mgr.smithi053.fgsmol (mgr.14164) 130 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:30.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:30.130+0000 7f7deeaa8700 1 -- 172.21.15.146:0/84436132 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7de80f6510 msgr2=0x7f7de80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:30.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:30.130+0000 7f7deeaa8700 1 --2- 172.21.15.146:0/84436132 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7de80f6510 0x7f7de80f6930 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f7ddc009fb0 tx=0x7f7ddc009b30).stop 2024-01-21T20:03:30.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:30.130+0000 7f7deeaa8700 1 -- 172.21.15.146:0/84436132 shutdown_connections 2024-01-21T20:03:30.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:30.130+0000 7f7deeaa8700 1 --2- 172.21.15.146:0/84436132 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7de80f6510 0x7f7de80f6930 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:30.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:30.130+0000 7f7deeaa8700 1 -- 172.21.15.146:0/84436132 >> 172.21.15.146:0/84436132 conn(0x7f7de80f1c70 msgr2=0x7f7de80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:30.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:30.130+0000 7f7deeaa8700 1 -- 172.21.15.146:0/84436132 shutdown_connections 2024-01-21T20:03:30.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:30.130+0000 7f7deeaa8700 1 -- 172.21.15.146:0/84436132 wait complete. 2024-01-21T20:03:30.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:30.130+0000 7f7deeaa8700 1 Processor -- start 2024-01-21T20:03:30.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:30.130+0000 7f7deeaa8700 1 -- start start 2024-01-21T20:03:30.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:30.130+0000 7f7deeaa8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7de80f6510 0x7f7de806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:30.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:30.130+0000 7f7deeaa8700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de8069e20 con 0x7f7de80f6510 2024-01-21T20:03:30.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:30.131+0000 7f7dec844700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7de80f6510 0x7f7de806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:30.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:30.131+0000 7f7dec844700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7de80f6510 0x7f7de806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:34046/0 (socket says 172.21.15.146:34046) 2024-01-21T20:03:30.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:30.131+0000 7f7dec844700 1 -- 172.21.15.146:0/4132433840 learned_addr learned my addr 172.21.15.146:0/4132433840 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:03:30.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:30.131+0000 7f7dec844700 1 -- 172.21.15.146:0/4132433840 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ddc005040 con 0x7f7de80f6510 2024-01-21T20:03:30.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:30.131+0000 7f7dec844700 1 --2- 172.21.15.146:0/4132433840 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7de80f6510 0x7f7de806b9a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f7ddc013510 tx=0x7f7ddc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:30.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:30.131+0000 7f7de57fa700 1 -- 172.21.15.146:0/4132433840 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ddc009320 con 0x7f7de80f6510 2024-01-21T20:03:30.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:30.131+0000 7f7deeaa8700 1 -- 172.21.15.146:0/4132433840 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7de8069fc0 con 0x7f7de80f6510 2024-01-21T20:03:30.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:30.131+0000 7f7de57fa700 1 -- 172.21.15.146:0/4132433840 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7ddc013f00 con 0x7f7de80f6510 2024-01-21T20:03:30.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:30.131+0000 7f7de57fa700 1 -- 172.21.15.146:0/4132433840 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ddc014d90 con 0x7f7de80f6510 2024-01-21T20:03:30.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:30.131+0000 7f7deeaa8700 1 -- 172.21.15.146:0/4132433840 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7de806a450 con 0x7f7de80f6510 2024-01-21T20:03:30.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:30.132+0000 7f7de57fa700 1 -- 172.21.15.146:0/4132433840 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7ddc0095f0 con 0x7f7de80f6510 2024-01-21T20:03:30.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:30.132+0000 7f7deeaa8700 1 -- 172.21.15.146:0/4132433840 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7de8059100 con 0x7f7de80f6510 2024-01-21T20:03:30.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:30.132+0000 7f7de57fa700 1 --2- 172.21.15.146:0/4132433840 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f7dd802ff60 0x7f7dd8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:30.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:30.132+0000 7f7de57fa700 1 -- 172.21.15.146:0/4132433840 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7ddc040fe0 con 0x7f7de80f6510 2024-01-21T20:03:30.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:30.133+0000 7f7de7fff700 1 --2- 172.21.15.146:0/4132433840 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f7dd802ff60 0x7f7dd8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:30.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:30.133+0000 7f7de7fff700 1 --2- 172.21.15.146:0/4132433840 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f7dd802ff60 0x7f7dd8032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f7dd40096a0 tx=0x7f7dd4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:30.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:30.138+0000 7f7de57fa700 1 -- 172.21.15.146:0/4132433840 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7ddc01d7f0 con 0x7f7de80f6510 2024-01-21T20:03:30.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:30.343+0000 7f7deeaa8700 1 -- 172.21.15.146:0/4132433840 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7de80fbf60 con 0x7f7de80f6510 2024-01-21T20:03:30.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:30.344+0000 7f7de57fa700 1 -- 172.21.15.146:0/4132433840 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7ddc010230 con 0x7f7de80f6510 2024-01-21T20:03:30.335 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:03:30.335 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:03:30.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:30.345+0000 7f7deeaa8700 1 -- 172.21.15.146:0/4132433840 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f7dd802ff60 msgr2=0x7f7dd8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:30.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:30.345+0000 7f7deeaa8700 1 --2- 172.21.15.146:0/4132433840 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f7dd802ff60 0x7f7dd8032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f7dd40096a0 tx=0x7f7dd4006b40).stop 2024-01-21T20:03:30.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:30.345+0000 7f7deeaa8700 1 -- 172.21.15.146:0/4132433840 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7de80f6510 msgr2=0x7f7de806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:30.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:30.345+0000 7f7deeaa8700 1 --2- 172.21.15.146:0/4132433840 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7de80f6510 0x7f7de806b9a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f7ddc013510 tx=0x7f7ddc00b040).stop 2024-01-21T20:03:30.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:30.346+0000 7f7deeaa8700 1 -- 172.21.15.146:0/4132433840 shutdown_connections 2024-01-21T20:03:30.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:30.346+0000 7f7deeaa8700 1 --2- 172.21.15.146:0/4132433840 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f7dd802ff60 0x7f7dd8032420 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:30.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:30.346+0000 7f7deeaa8700 1 --2- 172.21.15.146:0/4132433840 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7de80f6510 0x7f7de806b9a0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:30.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:30.346+0000 7f7deeaa8700 1 -- 172.21.15.146:0/4132433840 >> 172.21.15.146:0/4132433840 conn(0x7f7de80f1c70 msgr2=0x7f7de80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:30.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:30.346+0000 7f7deeaa8700 1 -- 172.21.15.146:0/4132433840 shutdown_connections 2024-01-21T20:03:30.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:30.346+0000 7f7deeaa8700 1 -- 172.21.15.146:0/4132433840 wait complete. 2024-01-21T20:03:30.338 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:03:31.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:30 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:03:29.166577+0000 mgr.smithi053.fgsmol (mgr.14164) 131 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:31.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:30 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:03:30.345155+0000 mon.smithi053 (mon.0) 206 : audit [DBG] from='client.? 172.21.15.146:0/4132433840' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:03:31.762 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:03:31.762 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:03:32.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:32.994+0000 7fe8bb307700 1 -- 172.21.15.146:0/3632457270 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8b40fcbe0 msgr2=0x7fe8b40fd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:32.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:32.994+0000 7fe8bb307700 1 --2- 172.21.15.146:0/3632457270 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8b40fcbe0 0x7fe8b40fd000 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7fe8b0009fb0 tx=0x7fe8b0009b30).stop 2024-01-21T20:03:32.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:32.994+0000 7fe8bb307700 1 -- 172.21.15.146:0/3632457270 shutdown_connections 2024-01-21T20:03:32.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:32.994+0000 7fe8bb307700 1 --2- 172.21.15.146:0/3632457270 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8b40fcbe0 0x7fe8b40fd000 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:32.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:32.994+0000 7fe8bb307700 1 -- 172.21.15.146:0/3632457270 >> 172.21.15.146:0/3632457270 conn(0x7fe8b406be10 msgr2=0x7fe8b406e230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:32.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:32.994+0000 7fe8bb307700 1 -- 172.21.15.146:0/3632457270 shutdown_connections 2024-01-21T20:03:32.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:32.994+0000 7fe8bb307700 1 -- 172.21.15.146:0/3632457270 wait complete. 2024-01-21T20:03:32.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:32.995+0000 7fe8bb307700 1 Processor -- start 2024-01-21T20:03:32.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:32.995+0000 7fe8bb307700 1 -- start start 2024-01-21T20:03:32.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:32.995+0000 7fe8bb307700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8b40fcbe0 0x7fe8b410cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:32.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:32.995+0000 7fe8bb307700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b410d230 con 0x7fe8b40fcbe0 2024-01-21T20:03:32.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:32.995+0000 7fe8b90a3700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8b40fcbe0 0x7fe8b410cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:32.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:32.995+0000 7fe8b90a3700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8b40fcbe0 0x7fe8b410cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:34056/0 (socket says 172.21.15.146:34056) 2024-01-21T20:03:32.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:32.995+0000 7fe8b90a3700 1 -- 172.21.15.146:0/1650031192 learned_addr learned my addr 172.21.15.146:0/1650031192 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:03:32.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:32.996+0000 7fe8b90a3700 1 -- 172.21.15.146:0/1650031192 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8b0005040 con 0x7fe8b40fcbe0 2024-01-21T20:03:32.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:32.996+0000 7fe8b90a3700 1 --2- 172.21.15.146:0/1650031192 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8b40fcbe0 0x7fe8b410cd40 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7fe8b0013510 tx=0x7fe8b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:32.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:32.996+0000 7fe8a9ffb700 1 -- 172.21.15.146:0/1650031192 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8b0009320 con 0x7fe8b40fcbe0 2024-01-21T20:03:32.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:32.996+0000 7fe8bb307700 1 -- 172.21.15.146:0/1650031192 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8b410d430 con 0x7fe8b40fcbe0 2024-01-21T20:03:32.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:32.996+0000 7fe8a9ffb700 1 -- 172.21.15.146:0/1650031192 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe8b0013f00 con 0x7fe8b40fcbe0 2024-01-21T20:03:32.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:32.996+0000 7fe8a9ffb700 1 -- 172.21.15.146:0/1650031192 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8b0014dc0 con 0x7fe8b40fcbe0 2024-01-21T20:03:32.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:32.996+0000 7fe8bb307700 1 -- 172.21.15.146:0/1650031192 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8b410dfe0 con 0x7fe8b40fcbe0 2024-01-21T20:03:32.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:32.997+0000 7fe8a9ffb700 1 -- 172.21.15.146:0/1650031192 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe8b00095f0 con 0x7fe8b40fcbe0 2024-01-21T20:03:32.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:32.997+0000 7fe8bb307700 1 -- 172.21.15.146:0/1650031192 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8b4059100 con 0x7fe8b40fcbe0 2024-01-21T20:03:32.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:32.997+0000 7fe8a9ffb700 1 --2- 172.21.15.146:0/1650031192 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fe8a002ffb0 0x7fe8a0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:32.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:32.997+0000 7fe8a9ffb700 1 -- 172.21.15.146:0/1650031192 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe8b0040f90 con 0x7fe8b40fcbe0 2024-01-21T20:03:32.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:32.997+0000 7fe8b88a2700 1 --2- 172.21.15.146:0/1650031192 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fe8a002ffb0 0x7fe8a0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:32.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:32.998+0000 7fe8b88a2700 1 --2- 172.21.15.146:0/1650031192 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fe8a002ffb0 0x7fe8a0032470 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fe8a40096a0 tx=0x7fe8a4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:32.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:33.002+0000 7fe8a9ffb700 1 -- 172.21.15.146:0/1650031192 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe8b0014420 con 0x7fe8b40fcbe0 2024-01-21T20:03:33.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:32 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:03:31.166895+0000 mgr.smithi053.fgsmol (mgr.14164) 132 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:33.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:33.207+0000 7fe8bb307700 1 -- 172.21.15.146:0/1650031192 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe8b410e700 con 0x7fe8b40fcbe0 2024-01-21T20:03:33.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:33.208+0000 7fe8a9ffb700 1 -- 172.21.15.146:0/1650031192 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe8b0010230 con 0x7fe8b40fcbe0 2024-01-21T20:03:33.198 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:03:33.198 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:03:33.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:33.210+0000 7fe8bb307700 1 -- 172.21.15.146:0/1650031192 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fe8a002ffb0 msgr2=0x7fe8a0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:33.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:33.210+0000 7fe8bb307700 1 --2- 172.21.15.146:0/1650031192 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fe8a002ffb0 0x7fe8a0032470 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fe8a40096a0 tx=0x7fe8a4006b40).stop 2024-01-21T20:03:33.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:33.210+0000 7fe8bb307700 1 -- 172.21.15.146:0/1650031192 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8b40fcbe0 msgr2=0x7fe8b410cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:33.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:33.210+0000 7fe8bb307700 1 --2- 172.21.15.146:0/1650031192 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8b40fcbe0 0x7fe8b410cd40 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7fe8b0013510 tx=0x7fe8b000b040).stop 2024-01-21T20:03:33.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:33.210+0000 7fe8bb307700 1 -- 172.21.15.146:0/1650031192 shutdown_connections 2024-01-21T20:03:33.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:33.210+0000 7fe8bb307700 1 --2- 172.21.15.146:0/1650031192 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fe8a002ffb0 0x7fe8a0032470 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:33.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:33.210+0000 7fe8bb307700 1 --2- 172.21.15.146:0/1650031192 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8b40fcbe0 0x7fe8b410cd40 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:33.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:33.210+0000 7fe8bb307700 1 -- 172.21.15.146:0/1650031192 >> 172.21.15.146:0/1650031192 conn(0x7fe8b406be10 msgr2=0x7fe8b406d910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:33.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:33.210+0000 7fe8bb307700 1 -- 172.21.15.146:0/1650031192 shutdown_connections 2024-01-21T20:03:33.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:33.210+0000 7fe8bb307700 1 -- 172.21.15.146:0/1650031192 wait complete. 2024-01-21T20:03:33.201 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:03:34.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:33 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:03:33.208480+0000 mon.smithi053 (mon.0) 207 : audit [DBG] from='client.? 172.21.15.146:0/1650031192' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:03:34.616 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:03:34.616 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:03:35.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:34 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:03:33.167279+0000 mgr.smithi053.fgsmol (mgr.14164) 133 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:35.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:35.830+0000 7fbcdf0de700 1 -- 172.21.15.146:0/864661242 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbcd8060280 msgr2=0x7fbcd80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:35.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:35.830+0000 7fbcdf0de700 1 --2- 172.21.15.146:0/864661242 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbcd8060280 0x7fbcd80f90f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7fbccc009fb0 tx=0x7fbccc009b30).stop 2024-01-21T20:03:35.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:35.830+0000 7fbcdf0de700 1 -- 172.21.15.146:0/864661242 shutdown_connections 2024-01-21T20:03:35.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:35.830+0000 7fbcdf0de700 1 --2- 172.21.15.146:0/864661242 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbcd8060280 0x7fbcd80f90f0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:35.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:35.830+0000 7fbcdf0de700 1 -- 172.21.15.146:0/864661242 >> 172.21.15.146:0/864661242 conn(0x7fbcd80f0b70 msgr2=0x7fbcd80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:35.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:35.831+0000 7fbcdf0de700 1 -- 172.21.15.146:0/864661242 shutdown_connections 2024-01-21T20:03:35.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:35.831+0000 7fbcdf0de700 1 -- 172.21.15.146:0/864661242 wait complete. 2024-01-21T20:03:35.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:35.831+0000 7fbcdf0de700 1 Processor -- start 2024-01-21T20:03:35.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:35.831+0000 7fbcdf0de700 1 -- start start 2024-01-21T20:03:35.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:35.831+0000 7fbcdf0de700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbcd8060280 0x7fbcd81066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:35.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:35.831+0000 7fbcdf0de700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd8106bd0 con 0x7fbcd8060280 2024-01-21T20:03:35.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:35.832+0000 7fbcdce7a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbcd8060280 0x7fbcd81066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:35.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:35.832+0000 7fbcdce7a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbcd8060280 0x7fbcd81066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:34064/0 (socket says 172.21.15.146:34064) 2024-01-21T20:03:35.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:35.832+0000 7fbcdce7a700 1 -- 172.21.15.146:0/1230381407 learned_addr learned my addr 172.21.15.146:0/1230381407 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:03:35.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:35.832+0000 7fbcdce7a700 1 -- 172.21.15.146:0/1230381407 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbccc005040 con 0x7fbcd8060280 2024-01-21T20:03:35.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:35.832+0000 7fbcdce7a700 1 --2- 172.21.15.146:0/1230381407 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbcd8060280 0x7fbcd81066e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7fbccc013510 tx=0x7fbccc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:35.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:35.832+0000 7fbcd57fa700 1 -- 172.21.15.146:0/1230381407 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbccc009320 con 0x7fbcd8060280 2024-01-21T20:03:35.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:35.832+0000 7fbcdf0de700 1 -- 172.21.15.146:0/1230381407 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcd8106dd0 con 0x7fbcd8060280 2024-01-21T20:03:35.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:35.833+0000 7fbcd57fa700 1 -- 172.21.15.146:0/1230381407 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbccc013f00 con 0x7fbcd8060280 2024-01-21T20:03:35.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:35.833+0000 7fbcd57fa700 1 -- 172.21.15.146:0/1230381407 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbccc014dc0 con 0x7fbcd8060280 2024-01-21T20:03:35.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:35.833+0000 7fbcdf0de700 1 -- 172.21.15.146:0/1230381407 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcd8107260 con 0x7fbcd8060280 2024-01-21T20:03:35.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:35.833+0000 7fbcd57fa700 1 -- 172.21.15.146:0/1230381407 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbccc01d3d0 con 0x7fbcd8060280 2024-01-21T20:03:35.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:35.833+0000 7fbcd57fa700 1 --2- 172.21.15.146:0/1230381407 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fbcc802ff10 0x7fbcc80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:35.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:35.834+0000 7fbcd57fa700 1 -- 172.21.15.146:0/1230381407 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbccc0095f0 con 0x7fbcd8060280 2024-01-21T20:03:35.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:35.834+0000 7fbcdf0de700 1 -- 172.21.15.146:0/1230381407 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcd8045ad0 con 0x7fbcd8060280 2024-01-21T20:03:35.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:35.834+0000 7fbcd7fff700 1 --2- 172.21.15.146:0/1230381407 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fbcc802ff10 0x7fbcc80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:35.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:35.834+0000 7fbcd7fff700 1 --2- 172.21.15.146:0/1230381407 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fbcc802ff10 0x7fbcc80323d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fbcc40096a0 tx=0x7fbcc4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:35.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:35.838+0000 7fbcd57fa700 1 -- 172.21.15.146:0/1230381407 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbccc012920 con 0x7fbcd8060280 2024-01-21T20:03:36.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:36.043+0000 7fbcdf0de700 1 -- 172.21.15.146:0/1230381407 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbcd8107ee0 con 0x7fbcd8060280 2024-01-21T20:03:36.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:36.043+0000 7fbcd57fa700 1 -- 172.21.15.146:0/1230381407 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbccc012330 con 0x7fbcd8060280 2024-01-21T20:03:36.033 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:03:36.034 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:03:36.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:36.045+0000 7fbcdf0de700 1 -- 172.21.15.146:0/1230381407 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fbcc802ff10 msgr2=0x7fbcc80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:36.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:36.045+0000 7fbcdf0de700 1 --2- 172.21.15.146:0/1230381407 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fbcc802ff10 0x7fbcc80323d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fbcc40096a0 tx=0x7fbcc4006b40).stop 2024-01-21T20:03:36.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:36.045+0000 7fbcdf0de700 1 -- 172.21.15.146:0/1230381407 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbcd8060280 msgr2=0x7fbcd81066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:36.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:36.045+0000 7fbcdf0de700 1 --2- 172.21.15.146:0/1230381407 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbcd8060280 0x7fbcd81066e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7fbccc013510 tx=0x7fbccc00b040).stop 2024-01-21T20:03:36.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:36.045+0000 7fbcdf0de700 1 -- 172.21.15.146:0/1230381407 shutdown_connections 2024-01-21T20:03:36.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:36.045+0000 7fbcdf0de700 1 --2- 172.21.15.146:0/1230381407 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fbcc802ff10 0x7fbcc80323d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:36.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:36.045+0000 7fbcdf0de700 1 --2- 172.21.15.146:0/1230381407 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbcd8060280 0x7fbcd81066e0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:36.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:36.045+0000 7fbcdf0de700 1 -- 172.21.15.146:0/1230381407 >> 172.21.15.146:0/1230381407 conn(0x7fbcd80f0b70 msgr2=0x7fbcd80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:36.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:36.045+0000 7fbcdf0de700 1 -- 172.21.15.146:0/1230381407 shutdown_connections 2024-01-21T20:03:36.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:36.045+0000 7fbcdf0de700 1 -- 172.21.15.146:0/1230381407 wait complete. 2024-01-21T20:03:36.036 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:03:37.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:36 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:03:35.167621+0000 mgr.smithi053.fgsmol (mgr.14164) 134 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:37.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:36 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:03:36.043567+0000 mon.smithi053 (mon.0) 208 : audit [DBG] from='client.? 172.21.15.146:0/1230381407' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:03:37.472 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:03:37.472 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:03:38.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:38.674+0000 7f7944f7a700 1 -- 172.21.15.146:0/2916935862 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79400f5410 msgr2=0x7f79400f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:38.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:38.674+0000 7f7944f7a700 1 --2- 172.21.15.146:0/2916935862 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79400f5410 0x7f79400f5830 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f7930009fb0 tx=0x7f7930009b30).stop 2024-01-21T20:03:38.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:38.674+0000 7f7944f7a700 1 -- 172.21.15.146:0/2916935862 shutdown_connections 2024-01-21T20:03:38.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:38.674+0000 7f7944f7a700 1 --2- 172.21.15.146:0/2916935862 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79400f5410 0x7f79400f5830 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:38.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:38.674+0000 7f7944f7a700 1 -- 172.21.15.146:0/2916935862 >> 172.21.15.146:0/2916935862 conn(0x7f79400f0b70 msgr2=0x7f79400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:38.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:38.674+0000 7f7944f7a700 1 -- 172.21.15.146:0/2916935862 shutdown_connections 2024-01-21T20:03:38.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:38.674+0000 7f7944f7a700 1 -- 172.21.15.146:0/2916935862 wait complete. 2024-01-21T20:03:38.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:38.675+0000 7f7944f7a700 1 Processor -- start 2024-01-21T20:03:38.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:38.675+0000 7f7944f7a700 1 -- start start 2024-01-21T20:03:38.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:38.675+0000 7f7944f7a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79400f5410 0x7f7940105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:38.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:38.675+0000 7f7944f7a700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7940105b20 con 0x7f79400f5410 2024-01-21T20:03:38.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:38.675+0000 7f793e59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79400f5410 0x7f7940105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:38.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:38.675+0000 7f793e59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79400f5410 0x7f7940105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:45412/0 (socket says 172.21.15.146:45412) 2024-01-21T20:03:38.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:38.675+0000 7f793e59c700 1 -- 172.21.15.146:0/1523899608 learned_addr learned my addr 172.21.15.146:0/1523899608 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:03:38.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:38.676+0000 7f793e59c700 1 -- 172.21.15.146:0/1523899608 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7930005040 con 0x7f79400f5410 2024-01-21T20:03:38.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:38.676+0000 7f793e59c700 1 --2- 172.21.15.146:0/1523899608 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79400f5410 0x7f7940105630 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f7930013510 tx=0x7f793000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:38.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:38.676+0000 7f7936ffd700 1 -- 172.21.15.146:0/1523899608 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7930009320 con 0x7f79400f5410 2024-01-21T20:03:38.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:38.676+0000 7f7944f7a700 1 -- 172.21.15.146:0/1523899608 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7940105d20 con 0x7f79400f5410 2024-01-21T20:03:38.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:38.676+0000 7f7936ffd700 1 -- 172.21.15.146:0/1523899608 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7930013f00 con 0x7f79400f5410 2024-01-21T20:03:38.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:38.676+0000 7f7936ffd700 1 -- 172.21.15.146:0/1523899608 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7930014d90 con 0x7f79400f5410 2024-01-21T20:03:38.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:38.676+0000 7f7944f7a700 1 -- 172.21.15.146:0/1523899608 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79401061b0 con 0x7f79400f5410 2024-01-21T20:03:38.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:38.677+0000 7f7936ffd700 1 -- 172.21.15.146:0/1523899608 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f79300095f0 con 0x7f79400f5410 2024-01-21T20:03:38.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:38.677+0000 7f7944f7a700 1 -- 172.21.15.146:0/1523899608 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7940045ad0 con 0x7f79400f5410 2024-01-21T20:03:38.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:38.677+0000 7f7936ffd700 1 --2- 172.21.15.146:0/1523899608 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f792c02ff60 0x7f792c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:38.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:38.677+0000 7f7936ffd700 1 -- 172.21.15.146:0/1523899608 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7930040fe0 con 0x7f79400f5410 2024-01-21T20:03:38.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:38.677+0000 7f793dd9b700 1 --2- 172.21.15.146:0/1523899608 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f792c02ff60 0x7f792c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:38.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:38.678+0000 7f793dd9b700 1 --2- 172.21.15.146:0/1523899608 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f792c02ff60 0x7f792c032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f79280096a0 tx=0x7f7928006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:38.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:38.682+0000 7f7936ffd700 1 -- 172.21.15.146:0/1523899608 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f793001d7f0 con 0x7f79400f5410 2024-01-21T20:03:38.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:38.886+0000 7f7944f7a700 1 -- 172.21.15.146:0/1523899608 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f79400fae60 con 0x7f79400f5410 2024-01-21T20:03:38.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:38.886+0000 7f7936ffd700 1 -- 172.21.15.146:0/1523899608 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7930010230 con 0x7f79400f5410 2024-01-21T20:03:38.877 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:03:38.877 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:03:38.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:38.888+0000 7f7944f7a700 1 -- 172.21.15.146:0/1523899608 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f792c02ff60 msgr2=0x7f792c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:38.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:38.888+0000 7f7944f7a700 1 --2- 172.21.15.146:0/1523899608 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f792c02ff60 0x7f792c032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f79280096a0 tx=0x7f7928006b40).stop 2024-01-21T20:03:38.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:38.888+0000 7f7944f7a700 1 -- 172.21.15.146:0/1523899608 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79400f5410 msgr2=0x7f7940105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:38.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:38.888+0000 7f7944f7a700 1 --2- 172.21.15.146:0/1523899608 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79400f5410 0x7f7940105630 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f7930013510 tx=0x7f793000b040).stop 2024-01-21T20:03:38.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:38.888+0000 7f7944f7a700 1 -- 172.21.15.146:0/1523899608 shutdown_connections 2024-01-21T20:03:38.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:38.888+0000 7f7944f7a700 1 --2- 172.21.15.146:0/1523899608 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f792c02ff60 0x7f792c032420 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:38.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:38.888+0000 7f7944f7a700 1 --2- 172.21.15.146:0/1523899608 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79400f5410 0x7f7940105630 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:38.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:38.888+0000 7f7944f7a700 1 -- 172.21.15.146:0/1523899608 >> 172.21.15.146:0/1523899608 conn(0x7f79400f0b70 msgr2=0x7f79400f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:38.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:38.888+0000 7f7944f7a700 1 -- 172.21.15.146:0/1523899608 shutdown_connections 2024-01-21T20:03:38.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:38.888+0000 7f7944f7a700 1 -- 172.21.15.146:0/1523899608 wait complete. 2024-01-21T20:03:38.879 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:03:39.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:38 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:03:37.167910+0000 mgr.smithi053.fgsmol (mgr.14164) 135 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:40.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:39 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:03:38.886917+0000 mon.smithi053 (mon.0) 209 : audit [DBG] from='client.? 172.21.15.146:0/1523899608' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:03:40.301 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:03:40.301 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:03:41.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:40 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:03:39.168160+0000 mgr.smithi053.fgsmol (mgr.14164) 136 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:41.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:41.512+0000 7fb30e00c700 1 -- 172.21.15.146:0/4166346411 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3080f4740 msgr2=0x7fb3080f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:41.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:41.512+0000 7fb30e00c700 1 --2- 172.21.15.146:0/4166346411 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3080f4740 0x7fb3080f4b60 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7fb2f0009fb0 tx=0x7fb2f0009b30).stop 2024-01-21T20:03:41.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:41.512+0000 7fb30e00c700 1 -- 172.21.15.146:0/4166346411 shutdown_connections 2024-01-21T20:03:41.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:41.512+0000 7fb30e00c700 1 --2- 172.21.15.146:0/4166346411 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3080f4740 0x7fb3080f4b60 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:41.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:41.512+0000 7fb30e00c700 1 -- 172.21.15.146:0/4166346411 >> 172.21.15.146:0/4166346411 conn(0x7fb3080f0260 msgr2=0x7fb3080f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:41.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:41.512+0000 7fb30e00c700 1 -- 172.21.15.146:0/4166346411 shutdown_connections 2024-01-21T20:03:41.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:41.512+0000 7fb30e00c700 1 -- 172.21.15.146:0/4166346411 wait complete. 2024-01-21T20:03:41.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:41.513+0000 7fb30e00c700 1 Processor -- start 2024-01-21T20:03:41.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:41.513+0000 7fb30e00c700 1 -- start start 2024-01-21T20:03:41.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:41.513+0000 7fb30e00c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3080f4740 0x7fb3080f72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:41.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:41.513+0000 7fb30e00c700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3080f5770 con 0x7fb3080f4740 2024-01-21T20:03:41.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:41.514+0000 7fb3077fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3080f4740 0x7fb3080f72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:41.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:41.514+0000 7fb3077fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3080f4740 0x7fb3080f72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:45444/0 (socket says 172.21.15.146:45444) 2024-01-21T20:03:41.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:41.514+0000 7fb3077fe700 1 -- 172.21.15.146:0/4231698291 learned_addr learned my addr 172.21.15.146:0/4231698291 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:03:41.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:41.514+0000 7fb3077fe700 1 -- 172.21.15.146:0/4231698291 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2f0005040 con 0x7fb3080f4740 2024-01-21T20:03:41.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:41.514+0000 7fb3077fe700 1 --2- 172.21.15.146:0/4231698291 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3080f4740 0x7fb3080f72f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fb2f0013510 tx=0x7fb2f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:41.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:41.514+0000 7fb2fffff700 1 -- 172.21.15.146:0/4231698291 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2f0009320 con 0x7fb3080f4740 2024-01-21T20:03:41.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:41.514+0000 7fb30e00c700 1 -- 172.21.15.146:0/4231698291 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3080f5910 con 0x7fb3080f4740 2024-01-21T20:03:41.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:41.514+0000 7fb2fffff700 1 -- 172.21.15.146:0/4231698291 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb2f0013f00 con 0x7fb3080f4740 2024-01-21T20:03:41.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:41.514+0000 7fb2fffff700 1 -- 172.21.15.146:0/4231698291 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2f0014d90 con 0x7fb3080f4740 2024-01-21T20:03:41.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:41.515+0000 7fb30e00c700 1 -- 172.21.15.146:0/4231698291 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3080f5da0 con 0x7fb3080f4740 2024-01-21T20:03:41.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:41.515+0000 7fb2fffff700 1 -- 172.21.15.146:0/4231698291 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb2f00263a0 con 0x7fb3080f4740 2024-01-21T20:03:41.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:41.515+0000 7fb2fffff700 1 --2- 172.21.15.146:0/4231698291 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb2f402ff60 0x7fb2f4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:41.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:41.515+0000 7fb2fffff700 1 -- 172.21.15.146:0/4231698291 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb2f0041300 con 0x7fb3080f4740 2024-01-21T20:03:41.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:41.516+0000 7fb306ffd700 1 --2- 172.21.15.146:0/4231698291 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb2f402ff60 0x7fb2f4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:41.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:41.516+0000 7fb306ffd700 1 --2- 172.21.15.146:0/4231698291 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb2f402ff60 0x7fb2f4032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fb2f80096a0 tx=0x7fb2f8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:41.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:41.516+0000 7fb30e00c700 1 -- 172.21.15.146:0/4231698291 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb308059100 con 0x7fb3080f4740 2024-01-21T20:03:41.511 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:41.521+0000 7fb2fffff700 1 -- 172.21.15.146:0/4231698291 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb2f001cb10 con 0x7fb3080f4740 2024-01-21T20:03:41.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:41.725+0000 7fb30e00c700 1 -- 172.21.15.146:0/4231698291 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb3080f6400 con 0x7fb3080f4740 2024-01-21T20:03:41.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:41.726+0000 7fb2fffff700 1 -- 172.21.15.146:0/4231698291 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb2f002b030 con 0x7fb3080f4740 2024-01-21T20:03:41.717 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:03:41.717 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:03:41.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:41.728+0000 7fb30e00c700 1 -- 172.21.15.146:0/4231698291 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb2f402ff60 msgr2=0x7fb2f4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:41.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:41.728+0000 7fb30e00c700 1 --2- 172.21.15.146:0/4231698291 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb2f402ff60 0x7fb2f4032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fb2f80096a0 tx=0x7fb2f8006b40).stop 2024-01-21T20:03:41.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:41.728+0000 7fb30e00c700 1 -- 172.21.15.146:0/4231698291 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3080f4740 msgr2=0x7fb3080f72f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:41.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:41.728+0000 7fb30e00c700 1 --2- 172.21.15.146:0/4231698291 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3080f4740 0x7fb3080f72f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fb2f0013510 tx=0x7fb2f000b040).stop 2024-01-21T20:03:41.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:41.728+0000 7fb30e00c700 1 -- 172.21.15.146:0/4231698291 shutdown_connections 2024-01-21T20:03:41.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:41.728+0000 7fb30e00c700 1 --2- 172.21.15.146:0/4231698291 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb2f402ff60 0x7fb2f4032420 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:41.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:41.728+0000 7fb30e00c700 1 --2- 172.21.15.146:0/4231698291 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3080f4740 0x7fb3080f72f0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:41.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:41.728+0000 7fb30e00c700 1 -- 172.21.15.146:0/4231698291 >> 172.21.15.146:0/4231698291 conn(0x7fb3080f0260 msgr2=0x7fb3080f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:41.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:41.728+0000 7fb30e00c700 1 -- 172.21.15.146:0/4231698291 shutdown_connections 2024-01-21T20:03:41.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:41.728+0000 7fb30e00c700 1 -- 172.21.15.146:0/4231698291 wait complete. 2024-01-21T20:03:41.719 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:03:42.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:42 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:03:41.168445+0000 mgr.smithi053.fgsmol (mgr.14164) 137 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:42.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:42 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:03:41.726549+0000 mon.smithi053 (mon.0) 210 : audit [DBG] from='client.? 172.21.15.146:0/4231698291' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:03:43.146 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:03:43.146 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:03:44.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:44.339+0000 7f877af56700 1 -- 172.21.15.146:0/4227184508 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87740f4740 msgr2=0x7f87740f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:44.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:44.339+0000 7f877af56700 1 --2- 172.21.15.146:0/4227184508 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87740f4740 0x7f87740f4b60 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f8768009fb0 tx=0x7f8768009b30).stop 2024-01-21T20:03:44.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:44.339+0000 7f877af56700 1 -- 172.21.15.146:0/4227184508 shutdown_connections 2024-01-21T20:03:44.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:44.339+0000 7f877af56700 1 --2- 172.21.15.146:0/4227184508 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87740f4740 0x7f87740f4b60 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:44.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:44.339+0000 7f877af56700 1 -- 172.21.15.146:0/4227184508 >> 172.21.15.146:0/4227184508 conn(0x7f87740f0260 msgr2=0x7f87740f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:44.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:44.339+0000 7f877af56700 1 -- 172.21.15.146:0/4227184508 shutdown_connections 2024-01-21T20:03:44.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:44.339+0000 7f877af56700 1 -- 172.21.15.146:0/4227184508 wait complete. 2024-01-21T20:03:44.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:44.340+0000 7f877af56700 1 Processor -- start 2024-01-21T20:03:44.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:44.340+0000 7f877af56700 1 -- start start 2024-01-21T20:03:44.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:44.340+0000 7f877af56700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87740f4740 0x7f87741066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:44.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:44.340+0000 7f877af56700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8774106bc0 con 0x7f87740f4740 2024-01-21T20:03:44.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:44.341+0000 7f8778cf2700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87740f4740 0x7f87741066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:44.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:44.341+0000 7f8778cf2700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87740f4740 0x7f87741066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:45466/0 (socket says 172.21.15.146:45466) 2024-01-21T20:03:44.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:44.341+0000 7f8778cf2700 1 -- 172.21.15.146:0/3498914611 learned_addr learned my addr 172.21.15.146:0/3498914611 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:03:44.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:44.341+0000 7f8778cf2700 1 -- 172.21.15.146:0/3498914611 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8768005040 con 0x7f87740f4740 2024-01-21T20:03:44.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:44.341+0000 7f8778cf2700 1 --2- 172.21.15.146:0/3498914611 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87740f4740 0x7f87741066d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f8768013510 tx=0x7f876800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:44.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:44.341+0000 7f87717fa700 1 -- 172.21.15.146:0/3498914611 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8768009320 con 0x7f87740f4740 2024-01-21T20:03:44.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:44.341+0000 7f877af56700 1 -- 172.21.15.146:0/3498914611 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8774106dc0 con 0x7f87740f4740 2024-01-21T20:03:44.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:44.341+0000 7f87717fa700 1 -- 172.21.15.146:0/3498914611 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8768013f00 con 0x7f87740f4740 2024-01-21T20:03:44.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:44.341+0000 7f87717fa700 1 -- 172.21.15.146:0/3498914611 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8768014d90 con 0x7f87740f4740 2024-01-21T20:03:44.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:44.341+0000 7f877af56700 1 -- 172.21.15.146:0/3498914611 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8774107250 con 0x7f87740f4740 2024-01-21T20:03:44.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:44.342+0000 7f87717fa700 1 -- 172.21.15.146:0/3498914611 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f87680263a0 con 0x7f87740f4740 2024-01-21T20:03:44.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:44.342+0000 7f87717fa700 1 --2- 172.21.15.146:0/3498914611 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f876402ff60 0x7f8764032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:44.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:44.342+0000 7f877af56700 1 -- 172.21.15.146:0/3498914611 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8774045ad0 con 0x7f87740f4740 2024-01-21T20:03:44.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:44.342+0000 7f87717fa700 1 -- 172.21.15.146:0/3498914611 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8768041020 con 0x7f87740f4740 2024-01-21T20:03:44.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:44.343+0000 7f8773fff700 1 --2- 172.21.15.146:0/3498914611 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f876402ff60 0x7f8764032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:44.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:44.343+0000 7f8773fff700 1 --2- 172.21.15.146:0/3498914611 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f876402ff60 0x7f8764032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f87600096a0 tx=0x7f8760006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:44.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:44.347+0000 7f87717fa700 1 -- 172.21.15.146:0/3498914611 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8768012920 con 0x7f87740f4740 2024-01-21T20:03:44.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:44.554+0000 7f877af56700 1 -- 172.21.15.146:0/3498914611 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f87741080a0 con 0x7f87740f4740 2024-01-21T20:03:44.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:44.555+0000 7f87717fa700 1 -- 172.21.15.146:0/3498914611 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f876802b020 con 0x7f87740f4740 2024-01-21T20:03:44.546 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:03:44.546 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:03:44.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:44.556+0000 7f877af56700 1 -- 172.21.15.146:0/3498914611 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f876402ff60 msgr2=0x7f8764032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:44.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:44.557+0000 7f877af56700 1 --2- 172.21.15.146:0/3498914611 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f876402ff60 0x7f8764032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f87600096a0 tx=0x7f8760006b40).stop 2024-01-21T20:03:44.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:44.557+0000 7f877af56700 1 -- 172.21.15.146:0/3498914611 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87740f4740 msgr2=0x7f87741066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:44.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:44.557+0000 7f877af56700 1 --2- 172.21.15.146:0/3498914611 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87740f4740 0x7f87741066d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f8768013510 tx=0x7f876800b040).stop 2024-01-21T20:03:44.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:44.557+0000 7f877af56700 1 -- 172.21.15.146:0/3498914611 shutdown_connections 2024-01-21T20:03:44.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:44.557+0000 7f877af56700 1 --2- 172.21.15.146:0/3498914611 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f876402ff60 0x7f8764032420 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:44.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:44.557+0000 7f877af56700 1 --2- 172.21.15.146:0/3498914611 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87740f4740 0x7f87741066d0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:44.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:44.557+0000 7f877af56700 1 -- 172.21.15.146:0/3498914611 >> 172.21.15.146:0/3498914611 conn(0x7f87740f0260 msgr2=0x7f87740f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:44.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:44.557+0000 7f877af56700 1 -- 172.21.15.146:0/3498914611 shutdown_connections 2024-01-21T20:03:44.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:44.557+0000 7f877af56700 1 -- 172.21.15.146:0/3498914611 wait complete. 2024-01-21T20:03:44.549 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:03:44.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:44 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:03:43.168723+0000 mgr.smithi053.fgsmol (mgr.14164) 138 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:45.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:45 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:03:44.556375+0000 mon.smithi053 (mon.0) 211 : audit [DBG] from='client.? 172.21.15.146:0/3498914611' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:03:45.979 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:03:45.979 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:03:46.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:46 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:03:45.169073+0000 mgr.smithi053.fgsmol (mgr.14164) 139 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:47.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:47.259+0000 7fa8fd919700 1 -- 172.21.15.146:0/1076431432 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8f80f78e0 msgr2=0x7fa8f80f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:47.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:47.259+0000 7fa8fd919700 1 --2- 172.21.15.146:0/1076431432 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8f80f78e0 0x7fa8f80f7d00 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7fa8ec009fb0 tx=0x7fa8ec009b30).stop 2024-01-21T20:03:47.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:47.259+0000 7fa8fd919700 1 -- 172.21.15.146:0/1076431432 shutdown_connections 2024-01-21T20:03:47.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:47.259+0000 7fa8fd919700 1 --2- 172.21.15.146:0/1076431432 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8f80f78e0 0x7fa8f80f7d00 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:47.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:47.259+0000 7fa8fd919700 1 -- 172.21.15.146:0/1076431432 >> 172.21.15.146:0/1076431432 conn(0x7fa8f80f0120 msgr2=0x7fa8f80f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:47.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:47.259+0000 7fa8fd919700 1 -- 172.21.15.146:0/1076431432 shutdown_connections 2024-01-21T20:03:47.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:47.259+0000 7fa8fd919700 1 -- 172.21.15.146:0/1076431432 wait complete. 2024-01-21T20:03:47.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:47.259+0000 7fa8fd919700 1 Processor -- start 2024-01-21T20:03:47.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:47.259+0000 7fa8fd919700 1 -- start start 2024-01-21T20:03:47.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:47.260+0000 7fa8fd919700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8f80f78e0 0x7fa8f81064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:47.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:47.260+0000 7fa8fd919700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f81069d0 con 0x7fa8f80f78e0 2024-01-21T20:03:47.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:47.260+0000 7fa8fc917700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8f80f78e0 0x7fa8f81064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:47.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:47.260+0000 7fa8fc917700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8f80f78e0 0x7fa8f81064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:45480/0 (socket says 172.21.15.146:45480) 2024-01-21T20:03:47.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:47.260+0000 7fa8fc917700 1 -- 172.21.15.146:0/2521901321 learned_addr learned my addr 172.21.15.146:0/2521901321 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:03:47.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:47.260+0000 7fa8fc917700 1 -- 172.21.15.146:0/2521901321 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8ec005040 con 0x7fa8f80f78e0 2024-01-21T20:03:47.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:47.260+0000 7fa8fc917700 1 --2- 172.21.15.146:0/2521901321 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8f80f78e0 0x7fa8f81064e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fa8ec013510 tx=0x7fa8ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:47.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:47.261+0000 7fa8f57fa700 1 -- 172.21.15.146:0/2521901321 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8ec009320 con 0x7fa8f80f78e0 2024-01-21T20:03:47.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:47.261+0000 7fa8fd919700 1 -- 172.21.15.146:0/2521901321 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8f8106bd0 con 0x7fa8f80f78e0 2024-01-21T20:03:47.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:47.261+0000 7fa8f57fa700 1 -- 172.21.15.146:0/2521901321 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa8ec013f00 con 0x7fa8f80f78e0 2024-01-21T20:03:47.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:47.261+0000 7fa8f57fa700 1 -- 172.21.15.146:0/2521901321 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8ec014dc0 con 0x7fa8f80f78e0 2024-01-21T20:03:47.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:47.261+0000 7fa8fd919700 1 -- 172.21.15.146:0/2521901321 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8f8107060 con 0x7fa8f80f78e0 2024-01-21T20:03:47.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:47.261+0000 7fa8f57fa700 1 -- 172.21.15.146:0/2521901321 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa8ec0095f0 con 0x7fa8f80f78e0 2024-01-21T20:03:47.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:47.262+0000 7fa8fd919700 1 -- 172.21.15.146:0/2521901321 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8f8045ad0 con 0x7fa8f80f78e0 2024-01-21T20:03:47.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:47.262+0000 7fa8f57fa700 1 --2- 172.21.15.146:0/2521901321 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa8e802ff60 0x7fa8e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:47.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:47.262+0000 7fa8f57fa700 1 -- 172.21.15.146:0/2521901321 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa8ec041040 con 0x7fa8f80f78e0 2024-01-21T20:03:47.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:47.262+0000 7fa8f7fff700 1 --2- 172.21.15.146:0/2521901321 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa8e802ff60 0x7fa8e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:47.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:47.262+0000 7fa8f7fff700 1 --2- 172.21.15.146:0/2521901321 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa8e802ff60 0x7fa8e8032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fa8e40096a0 tx=0x7fa8e4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:47.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:47.267+0000 7fa8f57fa700 1 -- 172.21.15.146:0/2521901321 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa8ec014420 con 0x7fa8f80f78e0 2024-01-21T20:03:47.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:47.470+0000 7fa8fd919700 1 -- 172.21.15.146:0/2521901321 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa8f8107f10 con 0x7fa8f80f78e0 2024-01-21T20:03:47.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:47.471+0000 7fa8f57fa700 1 -- 172.21.15.146:0/2521901321 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa8ec010230 con 0x7fa8f80f78e0 2024-01-21T20:03:47.462 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:03:47.462 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:03:47.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:47.472+0000 7fa8fd919700 1 -- 172.21.15.146:0/2521901321 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa8e802ff60 msgr2=0x7fa8e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:47.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:47.472+0000 7fa8fd919700 1 --2- 172.21.15.146:0/2521901321 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa8e802ff60 0x7fa8e8032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fa8e40096a0 tx=0x7fa8e4006b40).stop 2024-01-21T20:03:47.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:47.472+0000 7fa8fd919700 1 -- 172.21.15.146:0/2521901321 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8f80f78e0 msgr2=0x7fa8f81064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:47.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:47.472+0000 7fa8fd919700 1 --2- 172.21.15.146:0/2521901321 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8f80f78e0 0x7fa8f81064e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fa8ec013510 tx=0x7fa8ec00b040).stop 2024-01-21T20:03:47.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:47.472+0000 7fa8fd919700 1 -- 172.21.15.146:0/2521901321 shutdown_connections 2024-01-21T20:03:47.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:47.472+0000 7fa8fd919700 1 --2- 172.21.15.146:0/2521901321 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa8e802ff60 0x7fa8e8032420 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:47.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:47.472+0000 7fa8fd919700 1 --2- 172.21.15.146:0/2521901321 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8f80f78e0 0x7fa8f81064e0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:47.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:47.472+0000 7fa8fd919700 1 -- 172.21.15.146:0/2521901321 >> 172.21.15.146:0/2521901321 conn(0x7fa8f80f0120 msgr2=0x7fa8f80f0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:47.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:47.472+0000 7fa8fd919700 1 -- 172.21.15.146:0/2521901321 shutdown_connections 2024-01-21T20:03:47.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:47.472+0000 7fa8fd919700 1 -- 172.21.15.146:0/2521901321 wait complete. 2024-01-21T20:03:47.465 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:03:48.854 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:03:48.854 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:03:48.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:48 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:03:47.169430+0000 mgr.smithi053.fgsmol (mgr.14164) 140 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:48.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:48 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:03:47.472021+0000 mon.smithi053 (mon.0) 212 : audit [DBG] from='client.? 172.21.15.146:0/2521901321' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:03:50.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:50.072+0000 7f7ac646d700 1 -- 172.21.15.146:0/1281243965 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ac00f51a0 msgr2=0x7f7ac00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:50.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:50.072+0000 7f7ac646d700 1 --2- 172.21.15.146:0/1281243965 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ac00f51a0 0x7f7ac00f55c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f7abc009fb0 tx=0x7f7abc009b30).stop 2024-01-21T20:03:50.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:50.072+0000 7f7ac646d700 1 -- 172.21.15.146:0/1281243965 shutdown_connections 2024-01-21T20:03:50.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:50.072+0000 7f7ac646d700 1 --2- 172.21.15.146:0/1281243965 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ac00f51a0 0x7f7ac00f55c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:50.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:50.072+0000 7f7ac646d700 1 -- 172.21.15.146:0/1281243965 >> 172.21.15.146:0/1281243965 conn(0x7f7ac00f09a0 msgr2=0x7f7ac00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:50.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:50.072+0000 7f7ac646d700 1 -- 172.21.15.146:0/1281243965 shutdown_connections 2024-01-21T20:03:50.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:50.072+0000 7f7ac646d700 1 -- 172.21.15.146:0/1281243965 wait complete. 2024-01-21T20:03:50.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:50.072+0000 7f7ac646d700 1 Processor -- start 2024-01-21T20:03:50.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:50.072+0000 7f7ac646d700 1 -- start start 2024-01-21T20:03:50.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:50.073+0000 7f7ac646d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ac00f51a0 0x7f7ac01064d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:50.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:50.073+0000 7f7ac646d700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac01069c0 con 0x7f7ac00f51a0 2024-01-21T20:03:50.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:50.073+0000 7f7ac546b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ac00f51a0 0x7f7ac01064d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:50.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:50.073+0000 7f7ac546b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ac00f51a0 0x7f7ac01064d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:42120/0 (socket says 172.21.15.146:42120) 2024-01-21T20:03:50.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:50.073+0000 7f7ac546b700 1 -- 172.21.15.146:0/2542231906 learned_addr learned my addr 172.21.15.146:0/2542231906 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:03:50.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:50.073+0000 7f7ac546b700 1 -- 172.21.15.146:0/2542231906 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7abc005040 con 0x7f7ac00f51a0 2024-01-21T20:03:50.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:50.073+0000 7f7ac546b700 1 --2- 172.21.15.146:0/2542231906 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ac00f51a0 0x7f7ac01064d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f7abc013510 tx=0x7f7abc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:50.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:50.073+0000 7f7ab5ffb700 1 -- 172.21.15.146:0/2542231906 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7abc009320 con 0x7f7ac00f51a0 2024-01-21T20:03:50.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:50.073+0000 7f7ab5ffb700 1 -- 172.21.15.146:0/2542231906 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7abc013f00 con 0x7f7ac00f51a0 2024-01-21T20:03:50.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:50.073+0000 7f7ac646d700 1 -- 172.21.15.146:0/2542231906 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ac0106bc0 con 0x7f7ac00f51a0 2024-01-21T20:03:50.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:50.073+0000 7f7ab5ffb700 1 -- 172.21.15.146:0/2542231906 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7abc014dc0 con 0x7f7ac00f51a0 2024-01-21T20:03:50.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:50.074+0000 7f7ac646d700 1 -- 172.21.15.146:0/2542231906 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ac0107050 con 0x7f7ac00f51a0 2024-01-21T20:03:50.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:50.074+0000 7f7ab5ffb700 1 -- 172.21.15.146:0/2542231906 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7abc01d3d0 con 0x7f7ac00f51a0 2024-01-21T20:03:50.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:50.075+0000 7f7ab5ffb700 1 --2- 172.21.15.146:0/2542231906 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f7aac02ff60 0x7f7aac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:50.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:50.075+0000 7f7ac646d700 1 -- 172.21.15.146:0/2542231906 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ac0045ad0 con 0x7f7ac00f51a0 2024-01-21T20:03:50.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:50.075+0000 7f7ab5ffb700 1 -- 172.21.15.146:0/2542231906 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7abc041030 con 0x7f7ac00f51a0 2024-01-21T20:03:50.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:50.075+0000 7f7ac4c6a700 1 --2- 172.21.15.146:0/2542231906 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f7aac02ff60 0x7f7aac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:50.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:50.075+0000 7f7ac4c6a700 1 --2- 172.21.15.146:0/2542231906 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f7aac02ff60 0x7f7aac032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f7ab00096a0 tx=0x7f7ab0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:50.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:50.079+0000 7f7ab5ffb700 1 -- 172.21.15.146:0/2542231906 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7abc012920 con 0x7f7ac00f51a0 2024-01-21T20:03:50.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:50.285+0000 7f7ac646d700 1 -- 172.21.15.146:0/2542231906 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7ac0107ea0 con 0x7f7ac00f51a0 2024-01-21T20:03:50.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:50.286+0000 7f7ab5ffb700 1 -- 172.21.15.146:0/2542231906 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7abc012330 con 0x7f7ac00f51a0 2024-01-21T20:03:50.277 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:03:50.277 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:03:50.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:50.287+0000 7f7ac646d700 1 -- 172.21.15.146:0/2542231906 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f7aac02ff60 msgr2=0x7f7aac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:50.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:50.287+0000 7f7ac646d700 1 --2- 172.21.15.146:0/2542231906 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f7aac02ff60 0x7f7aac032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f7ab00096a0 tx=0x7f7ab0006b40).stop 2024-01-21T20:03:50.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:50.287+0000 7f7ac646d700 1 -- 172.21.15.146:0/2542231906 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ac00f51a0 msgr2=0x7f7ac01064d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:50.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:50.287+0000 7f7ac646d700 1 --2- 172.21.15.146:0/2542231906 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ac00f51a0 0x7f7ac01064d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f7abc013510 tx=0x7f7abc00b040).stop 2024-01-21T20:03:50.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:50.287+0000 7f7ac646d700 1 -- 172.21.15.146:0/2542231906 shutdown_connections 2024-01-21T20:03:50.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:50.287+0000 7f7ac646d700 1 --2- 172.21.15.146:0/2542231906 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f7aac02ff60 0x7f7aac032420 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:50.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:50.287+0000 7f7ac646d700 1 --2- 172.21.15.146:0/2542231906 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ac00f51a0 0x7f7ac01064d0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:50.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:50.287+0000 7f7ac646d700 1 -- 172.21.15.146:0/2542231906 >> 172.21.15.146:0/2542231906 conn(0x7f7ac00f09a0 msgr2=0x7f7ac005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:50.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:50.287+0000 7f7ac646d700 1 -- 172.21.15.146:0/2542231906 shutdown_connections 2024-01-21T20:03:50.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:50.287+0000 7f7ac646d700 1 -- 172.21.15.146:0/2542231906 wait complete. 2024-01-21T20:03:50.279 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:03:50.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:50 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:03:49.169740+0000 mgr.smithi053.fgsmol (mgr.14164) 141 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:50.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:50 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:03:50.287125+0000 mon.smithi053 (mon.0) 213 : audit [DBG] from='client.? 172.21.15.146:0/2542231906' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:03:51.676 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:03:51.676 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:03:52.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:52.879+0000 7f3bdacfd700 1 -- 172.21.15.146:0/4220120817 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bd40f62a0 msgr2=0x7f3bd40f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:52.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:52.879+0000 7f3bdacfd700 1 --2- 172.21.15.146:0/4220120817 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bd40f62a0 0x7f3bd40f66c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f3bc4009fb0 tx=0x7f3bc4009b30).stop 2024-01-21T20:03:52.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:52.879+0000 7f3bdacfd700 1 -- 172.21.15.146:0/4220120817 shutdown_connections 2024-01-21T20:03:52.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:52.879+0000 7f3bdacfd700 1 --2- 172.21.15.146:0/4220120817 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bd40f62a0 0x7f3bd40f66c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:52.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:52.879+0000 7f3bdacfd700 1 -- 172.21.15.146:0/4220120817 >> 172.21.15.146:0/4220120817 conn(0x7f3bd40f1aa0 msgr2=0x7f3bd40f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:52.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:52.879+0000 7f3bdacfd700 1 -- 172.21.15.146:0/4220120817 shutdown_connections 2024-01-21T20:03:52.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:52.879+0000 7f3bdacfd700 1 -- 172.21.15.146:0/4220120817 wait complete. 2024-01-21T20:03:52.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:52.880+0000 7f3bdacfd700 1 Processor -- start 2024-01-21T20:03:52.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:52.880+0000 7f3bdacfd700 1 -- start start 2024-01-21T20:03:52.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:52.880+0000 7f3bdacfd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bd40f62a0 0x7f3bd41064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:52.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:52.880+0000 7f3bdacfd700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd41069a0 con 0x7f3bd40f62a0 2024-01-21T20:03:52.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:52.880+0000 7f3bd9cfb700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bd40f62a0 0x7f3bd41064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:52.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:52.880+0000 7f3bd9cfb700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bd40f62a0 0x7f3bd41064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:42126/0 (socket says 172.21.15.146:42126) 2024-01-21T20:03:52.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:52.880+0000 7f3bd9cfb700 1 -- 172.21.15.146:0/2881954416 learned_addr learned my addr 172.21.15.146:0/2881954416 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:03:52.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:52.880+0000 7f3bd9cfb700 1 -- 172.21.15.146:0/2881954416 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bc4005040 con 0x7f3bd40f62a0 2024-01-21T20:03:52.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:52.881+0000 7f3bd9cfb700 1 --2- 172.21.15.146:0/2881954416 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bd40f62a0 0x7f3bd41064b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f3bc4013510 tx=0x7f3bc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:52.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:52.881+0000 7f3bca7fc700 1 -- 172.21.15.146:0/2881954416 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3bc4009320 con 0x7f3bd40f62a0 2024-01-21T20:03:52.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:52.881+0000 7f3bca7fc700 1 -- 172.21.15.146:0/2881954416 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3bc4013f00 con 0x7f3bd40f62a0 2024-01-21T20:03:52.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:52.881+0000 7f3bdacfd700 1 -- 172.21.15.146:0/2881954416 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bd4106ba0 con 0x7f3bd40f62a0 2024-01-21T20:03:52.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:52.881+0000 7f3bca7fc700 1 -- 172.21.15.146:0/2881954416 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3bc4014dc0 con 0x7f3bd40f62a0 2024-01-21T20:03:52.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:52.881+0000 7f3bdacfd700 1 -- 172.21.15.146:0/2881954416 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bd4107030 con 0x7f3bd40f62a0 2024-01-21T20:03:52.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:52.882+0000 7f3bca7fc700 1 -- 172.21.15.146:0/2881954416 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3bc40095f0 con 0x7f3bd40f62a0 2024-01-21T20:03:52.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:52.882+0000 7f3bdacfd700 1 -- 172.21.15.146:0/2881954416 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bd4045ad0 con 0x7f3bd40f62a0 2024-01-21T20:03:52.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:52.882+0000 7f3bca7fc700 1 --2- 172.21.15.146:0/2881954416 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f3bc002ff60 0x7f3bc0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:52.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:52.882+0000 7f3bca7fc700 1 -- 172.21.15.146:0/2881954416 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3bc4041040 con 0x7f3bd40f62a0 2024-01-21T20:03:52.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:52.882+0000 7f3bd94fa700 1 --2- 172.21.15.146:0/2881954416 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f3bc002ff60 0x7f3bc0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:52.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:52.882+0000 7f3bd94fa700 1 --2- 172.21.15.146:0/2881954416 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f3bc002ff60 0x7f3bc0032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f3bd00096a0 tx=0x7f3bd0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:52.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:52.887+0000 7f3bca7fc700 1 -- 172.21.15.146:0/2881954416 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3bc4014420 con 0x7f3bd40f62a0 2024-01-21T20:03:52.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:52 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:03:51.169976+0000 mgr.smithi053.fgsmol (mgr.14164) 142 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:53.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:53.093+0000 7f3bdacfd700 1 -- 172.21.15.146:0/2881954416 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3bd4107ee0 con 0x7f3bd40f62a0 2024-01-21T20:03:53.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:53.094+0000 7f3bca7fc700 1 -- 172.21.15.146:0/2881954416 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3bc4010230 con 0x7f3bd40f62a0 2024-01-21T20:03:53.085 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:03:53.085 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:03:53.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:53.096+0000 7f3bdacfd700 1 -- 172.21.15.146:0/2881954416 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f3bc002ff60 msgr2=0x7f3bc0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:53.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:53.096+0000 7f3bdacfd700 1 --2- 172.21.15.146:0/2881954416 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f3bc002ff60 0x7f3bc0032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f3bd00096a0 tx=0x7f3bd0006b40).stop 2024-01-21T20:03:53.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:53.096+0000 7f3bdacfd700 1 -- 172.21.15.146:0/2881954416 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bd40f62a0 msgr2=0x7f3bd41064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:53.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:53.096+0000 7f3bdacfd700 1 --2- 172.21.15.146:0/2881954416 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bd40f62a0 0x7f3bd41064b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f3bc4013510 tx=0x7f3bc400b040).stop 2024-01-21T20:03:53.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:53.096+0000 7f3bdacfd700 1 -- 172.21.15.146:0/2881954416 shutdown_connections 2024-01-21T20:03:53.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:53.096+0000 7f3bdacfd700 1 --2- 172.21.15.146:0/2881954416 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f3bc002ff60 0x7f3bc0032420 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:53.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:53.096+0000 7f3bdacfd700 1 --2- 172.21.15.146:0/2881954416 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bd40f62a0 0x7f3bd41064b0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:53.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:53.096+0000 7f3bdacfd700 1 -- 172.21.15.146:0/2881954416 >> 172.21.15.146:0/2881954416 conn(0x7f3bd40f1aa0 msgr2=0x7f3bd40f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:53.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:53.096+0000 7f3bdacfd700 1 -- 172.21.15.146:0/2881954416 shutdown_connections 2024-01-21T20:03:53.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:53.096+0000 7f3bdacfd700 1 -- 172.21.15.146:0/2881954416 wait complete. 2024-01-21T20:03:53.088 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:03:53.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:53 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:03:53.095343+0000 mon.smithi053 (mon.0) 214 : audit [DBG] from='client.? 172.21.15.146:0/2881954416' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:03:54.512 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:03:54.512 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:03:54.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:54 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:03:53.170337+0000 mgr.smithi053.fgsmol (mgr.14164) 143 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:55.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:55.730+0000 7f6762cdf700 1 -- 172.21.15.146:0/2298394480 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f675c0f51a0 msgr2=0x7f675c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:55.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:55.730+0000 7f6762cdf700 1 --2- 172.21.15.146:0/2298394480 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f675c0f51a0 0x7f675c0f55c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f6758009fb0 tx=0x7f6758009b30).stop 2024-01-21T20:03:55.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:55.730+0000 7f6762cdf700 1 -- 172.21.15.146:0/2298394480 shutdown_connections 2024-01-21T20:03:55.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:55.730+0000 7f6762cdf700 1 --2- 172.21.15.146:0/2298394480 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f675c0f51a0 0x7f675c0f55c0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:55.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:55.730+0000 7f6762cdf700 1 -- 172.21.15.146:0/2298394480 >> 172.21.15.146:0/2298394480 conn(0x7f675c0f09a0 msgr2=0x7f675c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:55.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:55.730+0000 7f6762cdf700 1 -- 172.21.15.146:0/2298394480 shutdown_connections 2024-01-21T20:03:55.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:55.730+0000 7f6762cdf700 1 -- 172.21.15.146:0/2298394480 wait complete. 2024-01-21T20:03:55.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:55.731+0000 7f6762cdf700 1 Processor -- start 2024-01-21T20:03:55.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:55.731+0000 7f6762cdf700 1 -- start start 2024-01-21T20:03:55.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:55.731+0000 7f6762cdf700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f675c0f51a0 0x7f675c1091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:55.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:55.731+0000 7f6762cdf700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f675c060240 con 0x7f675c0f51a0 2024-01-21T20:03:55.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:55.731+0000 7f6761cdd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f675c0f51a0 0x7f675c1091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:55.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:55.731+0000 7f6761cdd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f675c0f51a0 0x7f675c1091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:42144/0 (socket says 172.21.15.146:42144) 2024-01-21T20:03:55.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:55.731+0000 7f6761cdd700 1 -- 172.21.15.146:0/3439132065 learned_addr learned my addr 172.21.15.146:0/3439132065 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:03:55.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:55.732+0000 7f6761cdd700 1 -- 172.21.15.146:0/3439132065 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6758005040 con 0x7f675c0f51a0 2024-01-21T20:03:55.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:55.732+0000 7f6761cdd700 1 --2- 172.21.15.146:0/3439132065 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f675c0f51a0 0x7f675c1091b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f6758013510 tx=0x7f675800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:55.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:55.732+0000 7f674e7fc700 1 -- 172.21.15.146:0/3439132065 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6758009320 con 0x7f675c0f51a0 2024-01-21T20:03:55.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:55.732+0000 7f6762cdf700 1 -- 172.21.15.146:0/3439132065 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f675c060440 con 0x7f675c0f51a0 2024-01-21T20:03:55.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:55.732+0000 7f674e7fc700 1 -- 172.21.15.146:0/3439132065 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6758013f00 con 0x7f675c0f51a0 2024-01-21T20:03:55.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:55.732+0000 7f674e7fc700 1 -- 172.21.15.146:0/3439132065 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6758014dc0 con 0x7f675c0f51a0 2024-01-21T20:03:55.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:55.732+0000 7f6762cdf700 1 -- 172.21.15.146:0/3439132065 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f675c109860 con 0x7f675c0f51a0 2024-01-21T20:03:55.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:55.733+0000 7f674e7fc700 1 -- 172.21.15.146:0/3439132065 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6758014420 con 0x7f675c0f51a0 2024-01-21T20:03:55.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:55.733+0000 7f674e7fc700 1 --2- 172.21.15.146:0/3439132065 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f674802ff60 0x7f6748032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:55.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:55.733+0000 7f6762cdf700 1 -- 172.21.15.146:0/3439132065 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f675c045ad0 con 0x7f675c0f51a0 2024-01-21T20:03:55.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:55.733+0000 7f674e7fc700 1 -- 172.21.15.146:0/3439132065 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6758041050 con 0x7f675c0f51a0 2024-01-21T20:03:55.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:55.733+0000 7f67614dc700 1 --2- 172.21.15.146:0/3439132065 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f674802ff60 0x7f6748032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:55.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:55.734+0000 7f67614dc700 1 --2- 172.21.15.146:0/3439132065 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f674802ff60 0x7f6748032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f67500096a0 tx=0x7f6750006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:55.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:55.738+0000 7f674e7fc700 1 -- 172.21.15.146:0/3439132065 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f67580095f0 con 0x7f675c0f51a0 2024-01-21T20:03:55.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:55.942+0000 7f6762cdf700 1 -- 172.21.15.146:0/3439132065 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f675c10a080 con 0x7f675c0f51a0 2024-01-21T20:03:55.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:55.943+0000 7f674e7fc700 1 -- 172.21.15.146:0/3439132065 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f675801cb10 con 0x7f675c0f51a0 2024-01-21T20:03:55.934 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:03:55.934 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:03:55.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:55.944+0000 7f6762cdf700 1 -- 172.21.15.146:0/3439132065 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f674802ff60 msgr2=0x7f6748032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:55.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:55.944+0000 7f6762cdf700 1 --2- 172.21.15.146:0/3439132065 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f674802ff60 0x7f6748032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f67500096a0 tx=0x7f6750006b40).stop 2024-01-21T20:03:55.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:55.944+0000 7f6762cdf700 1 -- 172.21.15.146:0/3439132065 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f675c0f51a0 msgr2=0x7f675c1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:55.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:55.944+0000 7f6762cdf700 1 --2- 172.21.15.146:0/3439132065 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f675c0f51a0 0x7f675c1091b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f6758013510 tx=0x7f675800b040).stop 2024-01-21T20:03:55.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:55.944+0000 7f6762cdf700 1 -- 172.21.15.146:0/3439132065 shutdown_connections 2024-01-21T20:03:55.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:55.944+0000 7f6762cdf700 1 --2- 172.21.15.146:0/3439132065 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f674802ff60 0x7f6748032420 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:55.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:55.944+0000 7f6762cdf700 1 --2- 172.21.15.146:0/3439132065 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f675c0f51a0 0x7f675c1091b0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:55.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:55.944+0000 7f6762cdf700 1 -- 172.21.15.146:0/3439132065 >> 172.21.15.146:0/3439132065 conn(0x7f675c0f09a0 msgr2=0x7f675c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:55.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:55.944+0000 7f6762cdf700 1 -- 172.21.15.146:0/3439132065 shutdown_connections 2024-01-21T20:03:55.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:55.944+0000 7f6762cdf700 1 -- 172.21.15.146:0/3439132065 wait complete. 2024-01-21T20:03:55.937 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:03:56.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:56 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:03:55.170654+0000 mgr.smithi053.fgsmol (mgr.14164) 144 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:56.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:56 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:03:55.944108+0000 mon.smithi053 (mon.0) 215 : audit [DBG] from='client.? 172.21.15.146:0/3439132065' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:03:57.362 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:03:57.362 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:03:58.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:58.500+0000 7f8ad68e2700 1 -- 172.21.15.146:0/3835417039 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ad00f47c0 msgr2=0x7f8ad00f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:58.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:58.500+0000 7f8ad68e2700 1 --2- 172.21.15.146:0/3835417039 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ad00f47c0 0x7f8ad00f4be0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f8ac4009fb0 tx=0x7f8ac4009b30).stop 2024-01-21T20:03:58.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:58.500+0000 7f8ad68e2700 1 -- 172.21.15.146:0/3835417039 shutdown_connections 2024-01-21T20:03:58.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:58.500+0000 7f8ad68e2700 1 --2- 172.21.15.146:0/3835417039 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ad00f47c0 0x7f8ad00f4be0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:58.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:58.500+0000 7f8ad68e2700 1 -- 172.21.15.146:0/3835417039 >> 172.21.15.146:0/3835417039 conn(0x7f8ad00f02e0 msgr2=0x7f8ad00f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:58.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:58.500+0000 7f8ad68e2700 1 -- 172.21.15.146:0/3835417039 shutdown_connections 2024-01-21T20:03:58.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:58.500+0000 7f8ad68e2700 1 -- 172.21.15.146:0/3835417039 wait complete. 2024-01-21T20:03:58.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:58.501+0000 7f8ad68e2700 1 Processor -- start 2024-01-21T20:03:58.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:58.501+0000 7f8ad68e2700 1 -- start start 2024-01-21T20:03:58.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:58.501+0000 7f8ad68e2700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ad00f47c0 0x7f8ad0106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:58.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:58.501+0000 7f8ad68e2700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad0106bf0 con 0x7f8ad00f47c0 2024-01-21T20:03:58.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:58.501+0000 7f8acffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ad00f47c0 0x7f8ad0106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:58.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:58.501+0000 7f8acffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ad00f47c0 0x7f8ad0106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:56552/0 (socket says 172.21.15.146:56552) 2024-01-21T20:03:58.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:58.501+0000 7f8acffff700 1 -- 172.21.15.146:0/2564646909 learned_addr learned my addr 172.21.15.146:0/2564646909 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:03:58.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:58.502+0000 7f8acffff700 1 -- 172.21.15.146:0/2564646909 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ac4005040 con 0x7f8ad00f47c0 2024-01-21T20:03:58.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:58.502+0000 7f8acffff700 1 --2- 172.21.15.146:0/2564646909 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ad00f47c0 0x7f8ad0106700 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f8ac4013510 tx=0x7f8ac400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:58.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:58.502+0000 7f8accff9700 1 -- 172.21.15.146:0/2564646909 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ac4009320 con 0x7f8ad00f47c0 2024-01-21T20:03:58.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:58.502+0000 7f8ad68e2700 1 -- 172.21.15.146:0/2564646909 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ad0106df0 con 0x7f8ad00f47c0 2024-01-21T20:03:58.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:58.502+0000 7f8ad68e2700 1 -- 172.21.15.146:0/2564646909 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ad0107280 con 0x7f8ad00f47c0 2024-01-21T20:03:58.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:58.502+0000 7f8accff9700 1 -- 172.21.15.146:0/2564646909 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8ac4013f00 con 0x7f8ad00f47c0 2024-01-21T20:03:58.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:58.502+0000 7f8accff9700 1 -- 172.21.15.146:0/2564646909 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ac4014d90 con 0x7f8ad00f47c0 2024-01-21T20:03:58.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:58.503+0000 7f8accff9700 1 -- 172.21.15.146:0/2564646909 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8ac40263a0 con 0x7f8ad00f47c0 2024-01-21T20:03:58.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:58.503+0000 7f8accff9700 1 --2- 172.21.15.146:0/2564646909 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f8ab802ff60 0x7f8ab8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:58.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:58.503+0000 7f8accff9700 1 -- 172.21.15.146:0/2564646909 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8ac4041070 con 0x7f8ad00f47c0 2024-01-21T20:03:58.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:58.503+0000 7f8ad68e2700 1 -- 172.21.15.146:0/2564646909 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ad0059100 con 0x7f8ad00f47c0 2024-01-21T20:03:58.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:58.503+0000 7f8acf7fe700 1 --2- 172.21.15.146:0/2564646909 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f8ab802ff60 0x7f8ab8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:58.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:58.504+0000 7f8acf7fe700 1 --2- 172.21.15.146:0/2564646909 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f8ab802ff60 0x7f8ab8032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f8ac00096a0 tx=0x7f8ac0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:58.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:58.508+0000 7f8accff9700 1 -- 172.21.15.146:0/2564646909 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8ac4012920 con 0x7f8ad00f47c0 2024-01-21T20:03:58.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:58.714+0000 7f8ad68e2700 1 -- 172.21.15.146:0/2564646909 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8ad0045ad0 con 0x7f8ad00f47c0 2024-01-21T20:03:58.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:58.714+0000 7f8accff9700 1 -- 172.21.15.146:0/2564646909 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8ac400f340 con 0x7f8ad00f47c0 2024-01-21T20:03:58.705 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:03:58.705 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:03:58.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:58.716+0000 7f8ad68e2700 1 -- 172.21.15.146:0/2564646909 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f8ab802ff60 msgr2=0x7f8ab8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:58.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:58.716+0000 7f8ad68e2700 1 --2- 172.21.15.146:0/2564646909 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f8ab802ff60 0x7f8ab8032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f8ac00096a0 tx=0x7f8ac0006b40).stop 2024-01-21T20:03:58.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:58.716+0000 7f8ad68e2700 1 -- 172.21.15.146:0/2564646909 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ad00f47c0 msgr2=0x7f8ad0106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:58.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:58.716+0000 7f8ad68e2700 1 --2- 172.21.15.146:0/2564646909 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ad00f47c0 0x7f8ad0106700 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f8ac4013510 tx=0x7f8ac400b040).stop 2024-01-21T20:03:58.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:58.716+0000 7f8ad68e2700 1 -- 172.21.15.146:0/2564646909 shutdown_connections 2024-01-21T20:03:58.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:58.716+0000 7f8ad68e2700 1 --2- 172.21.15.146:0/2564646909 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f8ab802ff60 0x7f8ab8032420 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:58.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:58.716+0000 7f8ad68e2700 1 --2- 172.21.15.146:0/2564646909 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ad00f47c0 0x7f8ad0106700 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:58.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:58.716+0000 7f8ad68e2700 1 -- 172.21.15.146:0/2564646909 >> 172.21.15.146:0/2564646909 conn(0x7f8ad00f02e0 msgr2=0x7f8ad00f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:58.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:58.716+0000 7f8ad68e2700 1 -- 172.21.15.146:0/2564646909 shutdown_connections 2024-01-21T20:03:58.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:03:58.716+0000 7f8ad68e2700 1 -- 172.21.15.146:0/2564646909 wait complete. 2024-01-21T20:03:58.708 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:03:58.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:58 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:03:57.170987+0000 mgr.smithi053.fgsmol (mgr.14164) 145 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:59.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:03:59 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:03:58.715525+0000 mon.smithi053 (mon.0) 216 : audit [DBG] from='client.? 172.21.15.146:0/2564646909' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:04:00.144 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:04:00.145 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:04:00.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:00 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:03:59.171338+0000 mgr.smithi053.fgsmol (mgr.14164) 146 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:01.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:01.303+0000 7f0d5316e700 1 -- 172.21.15.146:0/3122419882 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d4c0f5410 msgr2=0x7f0d4c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:01.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:01.303+0000 7f0d5316e700 1 --2- 172.21.15.146:0/3122419882 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d4c0f5410 0x7f0d4c0f5830 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f0d40009fb0 tx=0x7f0d40009b30).stop 2024-01-21T20:04:01.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:01.303+0000 7f0d5316e700 1 -- 172.21.15.146:0/3122419882 shutdown_connections 2024-01-21T20:04:01.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:01.303+0000 7f0d5316e700 1 --2- 172.21.15.146:0/3122419882 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d4c0f5410 0x7f0d4c0f5830 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:01.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:01.303+0000 7f0d5316e700 1 -- 172.21.15.146:0/3122419882 >> 172.21.15.146:0/3122419882 conn(0x7f0d4c0f0b70 msgr2=0x7f0d4c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:01.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:01.303+0000 7f0d5316e700 1 -- 172.21.15.146:0/3122419882 shutdown_connections 2024-01-21T20:04:01.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:01.303+0000 7f0d5316e700 1 -- 172.21.15.146:0/3122419882 wait complete. 2024-01-21T20:04:01.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:01.304+0000 7f0d5316e700 1 Processor -- start 2024-01-21T20:04:01.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:01.304+0000 7f0d5316e700 1 -- start start 2024-01-21T20:04:01.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:01.304+0000 7f0d5316e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d4c0f5410 0x7f0d4c1093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:01.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:01.304+0000 7f0d5316e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4c060220 con 0x7f0d4c0f5410 2024-01-21T20:04:01.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:01.305+0000 7f0d50f0a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d4c0f5410 0x7f0d4c1093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:01.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:01.305+0000 7f0d50f0a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d4c0f5410 0x7f0d4c1093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:56576/0 (socket says 172.21.15.146:56576) 2024-01-21T20:04:01.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:01.305+0000 7f0d50f0a700 1 -- 172.21.15.146:0/3650661354 learned_addr learned my addr 172.21.15.146:0/3650661354 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:04:01.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:01.305+0000 7f0d50f0a700 1 -- 172.21.15.146:0/3650661354 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d40005040 con 0x7f0d4c0f5410 2024-01-21T20:04:01.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:01.305+0000 7f0d50f0a700 1 --2- 172.21.15.146:0/3650661354 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d4c0f5410 0x7f0d4c1093f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f0d40014000 tx=0x7f0d4000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:01.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:01.305+0000 7f0d497fa700 1 -- 172.21.15.146:0/3650661354 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d400095f0 con 0x7f0d4c0f5410 2024-01-21T20:04:01.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:01.305+0000 7f0d5316e700 1 -- 172.21.15.146:0/3650661354 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d4c060420 con 0x7f0d4c0f5410 2024-01-21T20:04:01.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:01.305+0000 7f0d497fa700 1 -- 172.21.15.146:0/3650661354 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d40007e70 con 0x7f0d4c0f5410 2024-01-21T20:04:01.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:01.305+0000 7f0d497fa700 1 -- 172.21.15.146:0/3650661354 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d40014ea0 con 0x7f0d4c0f5410 2024-01-21T20:04:01.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:01.305+0000 7f0d5316e700 1 -- 172.21.15.146:0/3650661354 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d4c109aa0 con 0x7f0d4c0f5410 2024-01-21T20:04:01.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:01.306+0000 7f0d497fa700 1 -- 172.21.15.146:0/3650661354 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0d400079c0 con 0x7f0d4c0f5410 2024-01-21T20:04:01.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:01.306+0000 7f0d5316e700 1 -- 172.21.15.146:0/3650661354 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d4c059100 con 0x7f0d4c0f5410 2024-01-21T20:04:01.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:01.306+0000 7f0d497fa700 1 --2- 172.21.15.146:0/3650661354 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0d3c02ff60 0x7f0d3c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:01.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:01.306+0000 7f0d497fa700 1 -- 172.21.15.146:0/3650661354 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0d40040920 con 0x7f0d4c0f5410 2024-01-21T20:04:01.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:01.306+0000 7f0d4bfff700 1 --2- 172.21.15.146:0/3650661354 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0d3c02ff60 0x7f0d3c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:01.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:01.307+0000 7f0d4bfff700 1 --2- 172.21.15.146:0/3650661354 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0d3c02ff60 0x7f0d3c032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f0d380096a0 tx=0x7f0d38006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:01.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:01.311+0000 7f0d497fa700 1 -- 172.21.15.146:0/3650661354 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0d4000f950 con 0x7f0d4c0f5410 2024-01-21T20:04:01.511 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:01.520+0000 7f0d5316e700 1 -- 172.21.15.146:0/3650661354 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0d4c045ad0 con 0x7f0d4c0f5410 2024-01-21T20:04:01.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:01.521+0000 7f0d497fa700 1 -- 172.21.15.146:0/3650661354 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0d40010230 con 0x7f0d4c0f5410 2024-01-21T20:04:01.512 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:04:01.512 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:04:01.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:01.522+0000 7f0d5316e700 1 -- 172.21.15.146:0/3650661354 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0d3c02ff60 msgr2=0x7f0d3c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:01.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:01.522+0000 7f0d5316e700 1 --2- 172.21.15.146:0/3650661354 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0d3c02ff60 0x7f0d3c032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f0d380096a0 tx=0x7f0d38006b40).stop 2024-01-21T20:04:01.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:01.522+0000 7f0d5316e700 1 -- 172.21.15.146:0/3650661354 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d4c0f5410 msgr2=0x7f0d4c1093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:01.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:01.522+0000 7f0d5316e700 1 --2- 172.21.15.146:0/3650661354 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d4c0f5410 0x7f0d4c1093f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f0d40014000 tx=0x7f0d4000b040).stop 2024-01-21T20:04:01.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:01.522+0000 7f0d5316e700 1 -- 172.21.15.146:0/3650661354 shutdown_connections 2024-01-21T20:04:01.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:01.522+0000 7f0d5316e700 1 --2- 172.21.15.146:0/3650661354 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0d3c02ff60 0x7f0d3c032420 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:01.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:01.522+0000 7f0d5316e700 1 --2- 172.21.15.146:0/3650661354 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d4c0f5410 0x7f0d4c1093f0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:01.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:01.522+0000 7f0d5316e700 1 -- 172.21.15.146:0/3650661354 >> 172.21.15.146:0/3650661354 conn(0x7f0d4c0f0b70 msgr2=0x7f0d4c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:01.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:01.522+0000 7f0d5316e700 1 -- 172.21.15.146:0/3650661354 shutdown_connections 2024-01-21T20:04:01.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:01.522+0000 7f0d5316e700 1 -- 172.21.15.146:0/3650661354 wait complete. 2024-01-21T20:04:01.515 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:04:02.899 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:04:02.899 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:04:02.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:02 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:04:01.171671+0000 mgr.smithi053.fgsmol (mgr.14164) 147 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:02.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:02 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:04:01.521938+0000 mon.smithi053 (mon.0) 217 : audit [DBG] from='client.? 172.21.15.146:0/3650661354' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:04:04.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:04.178+0000 7fc7befeb700 1 -- 172.21.15.146:0/1142814282 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7b80f51a0 msgr2=0x7fc7b80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:04.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:04.178+0000 7fc7befeb700 1 --2- 172.21.15.146:0/1142814282 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7b80f51a0 0x7fc7b80f55c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7fc7a8009fb0 tx=0x7fc7a8009b30).stop 2024-01-21T20:04:04.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:04.179+0000 7fc7befeb700 1 -- 172.21.15.146:0/1142814282 shutdown_connections 2024-01-21T20:04:04.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:04.179+0000 7fc7befeb700 1 --2- 172.21.15.146:0/1142814282 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7b80f51a0 0x7fc7b80f55c0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:04.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:04.179+0000 7fc7befeb700 1 -- 172.21.15.146:0/1142814282 >> 172.21.15.146:0/1142814282 conn(0x7fc7b80f09a0 msgr2=0x7fc7b80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:04.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:04.179+0000 7fc7befeb700 1 -- 172.21.15.146:0/1142814282 shutdown_connections 2024-01-21T20:04:04.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:04.179+0000 7fc7befeb700 1 -- 172.21.15.146:0/1142814282 wait complete. 2024-01-21T20:04:04.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:04.179+0000 7fc7befeb700 1 Processor -- start 2024-01-21T20:04:04.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:04.179+0000 7fc7befeb700 1 -- start start 2024-01-21T20:04:04.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:04.179+0000 7fc7befeb700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7b80f51a0 0x7fc7b81091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:04.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:04.179+0000 7fc7befeb700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b8060240 con 0x7fc7b80f51a0 2024-01-21T20:04:04.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:04.180+0000 7fc7bdfe9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7b80f51a0 0x7fc7b81091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:04.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:04.180+0000 7fc7bdfe9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7b80f51a0 0x7fc7b81091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:56592/0 (socket says 172.21.15.146:56592) 2024-01-21T20:04:04.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:04.180+0000 7fc7bdfe9700 1 -- 172.21.15.146:0/1058450880 learned_addr learned my addr 172.21.15.146:0/1058450880 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:04:04.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:04.180+0000 7fc7bdfe9700 1 -- 172.21.15.146:0/1058450880 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a8005040 con 0x7fc7b80f51a0 2024-01-21T20:04:04.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:04.180+0000 7fc7bdfe9700 1 --2- 172.21.15.146:0/1058450880 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7b80f51a0 0x7fc7b81091b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7fc7a8013510 tx=0x7fc7a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:04.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:04.180+0000 7fc7ae7fc700 1 -- 172.21.15.146:0/1058450880 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7a8009320 con 0x7fc7b80f51a0 2024-01-21T20:04:04.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:04.180+0000 7fc7ae7fc700 1 -- 172.21.15.146:0/1058450880 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc7a8013f00 con 0x7fc7b80f51a0 2024-01-21T20:04:04.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:04.180+0000 7fc7befeb700 1 -- 172.21.15.146:0/1058450880 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7b8060440 con 0x7fc7b80f51a0 2024-01-21T20:04:04.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:04.180+0000 7fc7ae7fc700 1 -- 172.21.15.146:0/1058450880 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7a8014dc0 con 0x7fc7b80f51a0 2024-01-21T20:04:04.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:04.180+0000 7fc7befeb700 1 -- 172.21.15.146:0/1058450880 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7b8109860 con 0x7fc7b80f51a0 2024-01-21T20:04:04.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:04.181+0000 7fc7ae7fc700 1 -- 172.21.15.146:0/1058450880 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc7a8014420 con 0x7fc7b80f51a0 2024-01-21T20:04:04.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:04.181+0000 7fc7ae7fc700 1 --2- 172.21.15.146:0/1058450880 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fc7a402ff60 0x7fc7a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:04.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:04.181+0000 7fc7ae7fc700 1 -- 172.21.15.146:0/1058450880 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc7a8041110 con 0x7fc7b80f51a0 2024-01-21T20:04:04.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:04.181+0000 7fc7befeb700 1 -- 172.21.15.146:0/1058450880 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7b8045ad0 con 0x7fc7b80f51a0 2024-01-21T20:04:04.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:04.181+0000 7fc7bd7e8700 1 --2- 172.21.15.146:0/1058450880 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fc7a402ff60 0x7fc7a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:04.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:04.182+0000 7fc7bd7e8700 1 --2- 172.21.15.146:0/1058450880 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fc7a402ff60 0x7fc7a4032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fc7b40096a0 tx=0x7fc7b4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:04.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:04.186+0000 7fc7ae7fc700 1 -- 172.21.15.146:0/1058450880 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc7a80095f0 con 0x7fc7b80f51a0 2024-01-21T20:04:04.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:04.393+0000 7fc7befeb700 1 -- 172.21.15.146:0/1058450880 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc7b80fabe0 con 0x7fc7b80f51a0 2024-01-21T20:04:04.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:04.394+0000 7fc7ae7fc700 1 -- 172.21.15.146:0/1058450880 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc7a802b020 con 0x7fc7b80f51a0 2024-01-21T20:04:04.385 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:04:04.385 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:04:04.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:04.395+0000 7fc7befeb700 1 -- 172.21.15.146:0/1058450880 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fc7a402ff60 msgr2=0x7fc7a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:04.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:04.395+0000 7fc7befeb700 1 --2- 172.21.15.146:0/1058450880 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fc7a402ff60 0x7fc7a4032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fc7b40096a0 tx=0x7fc7b4006b40).stop 2024-01-21T20:04:04.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:04.395+0000 7fc7befeb700 1 -- 172.21.15.146:0/1058450880 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7b80f51a0 msgr2=0x7fc7b81091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:04.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:04.395+0000 7fc7befeb700 1 --2- 172.21.15.146:0/1058450880 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7b80f51a0 0x7fc7b81091b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7fc7a8013510 tx=0x7fc7a800b040).stop 2024-01-21T20:04:04.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:04.395+0000 7fc7befeb700 1 -- 172.21.15.146:0/1058450880 shutdown_connections 2024-01-21T20:04:04.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:04.395+0000 7fc7befeb700 1 --2- 172.21.15.146:0/1058450880 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fc7a402ff60 0x7fc7a4032420 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:04.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:04.395+0000 7fc7befeb700 1 --2- 172.21.15.146:0/1058450880 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7b80f51a0 0x7fc7b81091b0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:04.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:04.395+0000 7fc7befeb700 1 -- 172.21.15.146:0/1058450880 >> 172.21.15.146:0/1058450880 conn(0x7fc7b80f09a0 msgr2=0x7fc7b80f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:04.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:04.395+0000 7fc7befeb700 1 -- 172.21.15.146:0/1058450880 shutdown_connections 2024-01-21T20:04:04.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:04.395+0000 7fc7befeb700 1 -- 172.21.15.146:0/1058450880 wait complete. 2024-01-21T20:04:04.387 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:04:04.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:04 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:04:03.172000+0000 mgr.smithi053.fgsmol (mgr.14164) 148 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:04.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:04 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:04:04.395164+0000 mon.smithi053 (mon.0) 218 : audit [DBG] from='client.? 172.21.15.146:0/1058450880' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:04:05.813 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:04:05.813 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:04:06.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:06 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:04:05.172294+0000 mgr.smithi053.fgsmol (mgr.14164) 149 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:06.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:06.991+0000 7f75f1647700 1 -- 172.21.15.146:0/1127006131 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75ec0f62a0 msgr2=0x7f75ec0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:06.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:06.991+0000 7f75f1647700 1 --2- 172.21.15.146:0/1127006131 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75ec0f62a0 0x7f75ec0f66c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f75dc009fb0 tx=0x7f75dc009b30).stop 2024-01-21T20:04:06.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:06.991+0000 7f75f1647700 1 -- 172.21.15.146:0/1127006131 shutdown_connections 2024-01-21T20:04:06.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:06.991+0000 7f75f1647700 1 --2- 172.21.15.146:0/1127006131 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75ec0f62a0 0x7f75ec0f66c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:06.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:06.991+0000 7f75f1647700 1 -- 172.21.15.146:0/1127006131 >> 172.21.15.146:0/1127006131 conn(0x7f75ec0f1aa0 msgr2=0x7f75ec0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:06.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:06.991+0000 7f75f1647700 1 -- 172.21.15.146:0/1127006131 shutdown_connections 2024-01-21T20:04:06.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:06.991+0000 7f75f1647700 1 -- 172.21.15.146:0/1127006131 wait complete. 2024-01-21T20:04:06.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:06.991+0000 7f75f1647700 1 Processor -- start 2024-01-21T20:04:06.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:06.992+0000 7f75f1647700 1 -- start start 2024-01-21T20:04:06.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:06.992+0000 7f75f1647700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75ec0f62a0 0x7f75ec06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:06.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:06.992+0000 7f75f1647700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75ec069e20 con 0x7f75ec0f62a0 2024-01-21T20:04:06.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:06.992+0000 7f75ebfff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75ec0f62a0 0x7f75ec06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:06.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:06.992+0000 7f75ebfff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75ec0f62a0 0x7f75ec06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:56612/0 (socket says 172.21.15.146:56612) 2024-01-21T20:04:06.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:06.992+0000 7f75ebfff700 1 -- 172.21.15.146:0/421767716 learned_addr learned my addr 172.21.15.146:0/421767716 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:04:06.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:06.992+0000 7f75ebfff700 1 -- 172.21.15.146:0/421767716 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75dc005040 con 0x7f75ec0f62a0 2024-01-21T20:04:06.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:06.992+0000 7f75ebfff700 1 --2- 172.21.15.146:0/421767716 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75ec0f62a0 0x7f75ec06b9a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f75dc009830 tx=0x7f75dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:06.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:06.993+0000 7f75e8ff9700 1 -- 172.21.15.146:0/421767716 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75dc009320 con 0x7f75ec0f62a0 2024-01-21T20:04:06.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:06.993+0000 7f75e8ff9700 1 -- 172.21.15.146:0/421767716 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f75dc0136c0 con 0x7f75ec0f62a0 2024-01-21T20:04:06.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:06.993+0000 7f75f1647700 1 -- 172.21.15.146:0/421767716 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75ec069fc0 con 0x7f75ec0f62a0 2024-01-21T20:04:06.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:06.993+0000 7f75e8ff9700 1 -- 172.21.15.146:0/421767716 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75dc01ee90 con 0x7f75ec0f62a0 2024-01-21T20:04:06.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:06.993+0000 7f75f1647700 1 -- 172.21.15.146:0/421767716 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75ec06a450 con 0x7f75ec0f62a0 2024-01-21T20:04:06.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:06.993+0000 7f75e8ff9700 1 -- 172.21.15.146:0/421767716 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f75dc01e420 con 0x7f75ec0f62a0 2024-01-21T20:04:06.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:06.994+0000 7f75f1647700 1 -- 172.21.15.146:0/421767716 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75ec059100 con 0x7f75ec0f62a0 2024-01-21T20:04:06.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:06.994+0000 7f75e8ff9700 1 --2- 172.21.15.146:0/421767716 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f75d402ff10 0x7f75d40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:06.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:06.994+0000 7f75e8ff9700 1 -- 172.21.15.146:0/421767716 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f75dc040fe0 con 0x7f75ec0f62a0 2024-01-21T20:04:06.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:06.994+0000 7f75eb7fe700 1 --2- 172.21.15.146:0/421767716 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f75d402ff10 0x7f75d40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:06.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:06.994+0000 7f75eb7fe700 1 --2- 172.21.15.146:0/421767716 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f75d402ff10 0x7f75d40323d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f75e00096a0 tx=0x7f75e0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:06.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:06.999+0000 7f75e8ff9700 1 -- 172.21.15.146:0/421767716 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f75dc01c790 con 0x7f75ec0f62a0 2024-01-21T20:04:07.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:07.207+0000 7f75f1647700 1 -- 172.21.15.146:0/421767716 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f75ec0fbce0 con 0x7f75ec0f62a0 2024-01-21T20:04:07.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:07.208+0000 7f75e8ff9700 1 -- 172.21.15.146:0/421767716 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f75dc02c2d0 con 0x7f75ec0f62a0 2024-01-21T20:04:07.199 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:04:07.199 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:04:07.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:07.209+0000 7f75f1647700 1 -- 172.21.15.146:0/421767716 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f75d402ff10 msgr2=0x7f75d40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:07.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:07.209+0000 7f75f1647700 1 --2- 172.21.15.146:0/421767716 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f75d402ff10 0x7f75d40323d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f75e00096a0 tx=0x7f75e0006b40).stop 2024-01-21T20:04:07.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:07.209+0000 7f75f1647700 1 -- 172.21.15.146:0/421767716 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75ec0f62a0 msgr2=0x7f75ec06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:07.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:07.209+0000 7f75f1647700 1 --2- 172.21.15.146:0/421767716 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75ec0f62a0 0x7f75ec06b9a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f75dc009830 tx=0x7f75dc00b040).stop 2024-01-21T20:04:07.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:07.209+0000 7f75f1647700 1 -- 172.21.15.146:0/421767716 shutdown_connections 2024-01-21T20:04:07.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:07.209+0000 7f75f1647700 1 --2- 172.21.15.146:0/421767716 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f75d402ff10 0x7f75d40323d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:07.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:07.209+0000 7f75f1647700 1 --2- 172.21.15.146:0/421767716 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75ec0f62a0 0x7f75ec06b9a0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:07.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:07.209+0000 7f75f1647700 1 -- 172.21.15.146:0/421767716 >> 172.21.15.146:0/421767716 conn(0x7f75ec0f1aa0 msgr2=0x7f75ec0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:07.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:07.209+0000 7f75f1647700 1 -- 172.21.15.146:0/421767716 shutdown_connections 2024-01-21T20:04:07.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:07.209+0000 7f75f1647700 1 -- 172.21.15.146:0/421767716 wait complete. 2024-01-21T20:04:07.201 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:04:07.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:07 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:04:07.209085+0000 mon.smithi053 (mon.0) 219 : audit [DBG] from='client.? 172.21.15.146:0/421767716' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:04:08.593 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:04:08.593 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:04:08.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:08 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:04:07.172574+0000 mgr.smithi053.fgsmol (mgr.14164) 150 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:09.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:09.886+0000 7f7641ecc700 1 -- 172.21.15.146:0/1067821042 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f763c0f5410 msgr2=0x7f763c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:09.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:09.886+0000 7f7641ecc700 1 --2- 172.21.15.146:0/1067821042 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f763c0f5410 0x7f763c0f5830 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f7630009fb0 tx=0x7f7630009b30).stop 2024-01-21T20:04:09.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:09.886+0000 7f7641ecc700 1 -- 172.21.15.146:0/1067821042 shutdown_connections 2024-01-21T20:04:09.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:09.886+0000 7f7641ecc700 1 --2- 172.21.15.146:0/1067821042 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f763c0f5410 0x7f763c0f5830 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:09.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:09.886+0000 7f7641ecc700 1 -- 172.21.15.146:0/1067821042 >> 172.21.15.146:0/1067821042 conn(0x7f763c0f0b70 msgr2=0x7f763c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:09.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:09.886+0000 7f7641ecc700 1 -- 172.21.15.146:0/1067821042 shutdown_connections 2024-01-21T20:04:09.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:09.886+0000 7f7641ecc700 1 -- 172.21.15.146:0/1067821042 wait complete. 2024-01-21T20:04:09.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:09.887+0000 7f7641ecc700 1 Processor -- start 2024-01-21T20:04:09.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:09.887+0000 7f7641ecc700 1 -- start start 2024-01-21T20:04:09.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:09.887+0000 7f7641ecc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f763c0f5410 0x7f763c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:09.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:09.887+0000 7f7641ecc700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f763c105b20 con 0x7f763c0f5410 2024-01-21T20:04:09.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:09.887+0000 7f763b7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f763c0f5410 0x7f763c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:09.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:09.887+0000 7f763b7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f763c0f5410 0x7f763c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:58346/0 (socket says 172.21.15.146:58346) 2024-01-21T20:04:09.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:09.887+0000 7f763b7fe700 1 -- 172.21.15.146:0/127985849 learned_addr learned my addr 172.21.15.146:0/127985849 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:04:09.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:09.888+0000 7f763b7fe700 1 -- 172.21.15.146:0/127985849 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7630005040 con 0x7f763c0f5410 2024-01-21T20:04:09.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:09.888+0000 7f763b7fe700 1 --2- 172.21.15.146:0/127985849 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f763c0f5410 0x7f763c105630 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f7630014000 tx=0x7f763000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:09.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:09.888+0000 7f7623fff700 1 -- 172.21.15.146:0/127985849 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76300095f0 con 0x7f763c0f5410 2024-01-21T20:04:09.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:09.888+0000 7f7641ecc700 1 -- 172.21.15.146:0/127985849 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f763c105d20 con 0x7f763c0f5410 2024-01-21T20:04:09.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:09.888+0000 7f7623fff700 1 -- 172.21.15.146:0/127985849 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7630007e70 con 0x7f763c0f5410 2024-01-21T20:04:09.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:09.888+0000 7f7623fff700 1 -- 172.21.15.146:0/127985849 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7630014ea0 con 0x7f763c0f5410 2024-01-21T20:04:09.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:09.888+0000 7f7641ecc700 1 -- 172.21.15.146:0/127985849 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f763c1061b0 con 0x7f763c0f5410 2024-01-21T20:04:09.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:09.889+0000 7f7623fff700 1 -- 172.21.15.146:0/127985849 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7630027070 con 0x7f763c0f5410 2024-01-21T20:04:09.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:09.889+0000 7f7623fff700 1 --2- 172.21.15.146:0/127985849 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f762402ff60 0x7f7624032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:09.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:09.889+0000 7f7623fff700 1 -- 172.21.15.146:0/127985849 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f76300427c0 con 0x7f763c0f5410 2024-01-21T20:04:09.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:09.889+0000 7f7641ecc700 1 -- 172.21.15.146:0/127985849 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f763c045ad0 con 0x7f763c0f5410 2024-01-21T20:04:09.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:09.889+0000 7f763affd700 1 --2- 172.21.15.146:0/127985849 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f762402ff60 0x7f7624032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:09.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:09.890+0000 7f763affd700 1 --2- 172.21.15.146:0/127985849 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f762402ff60 0x7f7624032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f762c0096a0 tx=0x7f762c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:09.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:09.894+0000 7f7623fff700 1 -- 172.21.15.146:0/127985849 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f763001f3c0 con 0x7f763c0f5410 2024-01-21T20:04:09.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:09 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:04:09.267552+0000 mon.smithi053 (mon.0) 220 : audit [INF] from='mgr.14164 172.21.15.53:0/2331671680' entity='mgr.smithi053.fgsmol' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fgsmol/mirror_snapshot_schedule"}]: dispatch 2024-01-21T20:04:09.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:09 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:04:09.275810+0000 mon.smithi053 (mon.0) 221 : audit [INF] from='mgr.14164 172.21.15.53:0/2331671680' entity='mgr.smithi053.fgsmol' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fgsmol/trash_purge_schedule"}]: dispatch 2024-01-21T20:04:10.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:10.099+0000 7f7641ecc700 1 -- 172.21.15.146:0/127985849 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f763c106ed0 con 0x7f763c0f5410 2024-01-21T20:04:10.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:10.100+0000 7f7623fff700 1 -- 172.21.15.146:0/127985849 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f763001e320 con 0x7f763c0f5410 2024-01-21T20:04:10.091 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:04:10.091 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:04:10.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:10.102+0000 7f7641ecc700 1 -- 172.21.15.146:0/127985849 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f762402ff60 msgr2=0x7f7624032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:10.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:10.102+0000 7f7641ecc700 1 --2- 172.21.15.146:0/127985849 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f762402ff60 0x7f7624032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f762c0096a0 tx=0x7f762c006b40).stop 2024-01-21T20:04:10.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:10.102+0000 7f7641ecc700 1 -- 172.21.15.146:0/127985849 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f763c0f5410 msgr2=0x7f763c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:10.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:10.102+0000 7f7641ecc700 1 --2- 172.21.15.146:0/127985849 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f763c0f5410 0x7f763c105630 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f7630014000 tx=0x7f763000b040).stop 2024-01-21T20:04:10.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:10.102+0000 7f7641ecc700 1 -- 172.21.15.146:0/127985849 shutdown_connections 2024-01-21T20:04:10.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:10.102+0000 7f7641ecc700 1 --2- 172.21.15.146:0/127985849 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f762402ff60 0x7f7624032420 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:10.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:10.102+0000 7f7641ecc700 1 --2- 172.21.15.146:0/127985849 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f763c0f5410 0x7f763c105630 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:10.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:10.102+0000 7f7641ecc700 1 -- 172.21.15.146:0/127985849 >> 172.21.15.146:0/127985849 conn(0x7f763c0f0b70 msgr2=0x7f763c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:10.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:10.102+0000 7f7641ecc700 1 -- 172.21.15.146:0/127985849 shutdown_connections 2024-01-21T20:04:10.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:10.102+0000 7f7641ecc700 1 -- 172.21.15.146:0/127985849 wait complete. 2024-01-21T20:04:10.094 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:04:10.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:10 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:04:09.172922+0000 mgr.smithi053.fgsmol (mgr.14164) 151 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:10.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:10 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:04:10.101341+0000 mon.smithi053 (mon.0) 222 : audit [DBG] from='client.? 172.21.15.146:0/127985849' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:04:11.472 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:04:11.472 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:04:12.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:12.734+0000 7f0b38258700 1 -- 172.21.15.146:0/3032675204 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0b300f6510 msgr2=0x7f0b300f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:12.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:12.734+0000 7f0b38258700 1 --2- 172.21.15.146:0/3032675204 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0b300f6510 0x7f0b300f6930 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f0b20009fb0 tx=0x7f0b20009b30).stop 2024-01-21T20:04:12.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:12.734+0000 7f0b38258700 1 -- 172.21.15.146:0/3032675204 shutdown_connections 2024-01-21T20:04:12.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:12.734+0000 7f0b38258700 1 --2- 172.21.15.146:0/3032675204 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0b300f6510 0x7f0b300f6930 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:12.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:12.734+0000 7f0b38258700 1 -- 172.21.15.146:0/3032675204 >> 172.21.15.146:0/3032675204 conn(0x7f0b300f1c70 msgr2=0x7f0b300f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:12.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:12.734+0000 7f0b38258700 1 -- 172.21.15.146:0/3032675204 shutdown_connections 2024-01-21T20:04:12.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:12.734+0000 7f0b38258700 1 -- 172.21.15.146:0/3032675204 wait complete. 2024-01-21T20:04:12.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:12.735+0000 7f0b38258700 1 Processor -- start 2024-01-21T20:04:12.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:12.735+0000 7f0b38258700 1 -- start start 2024-01-21T20:04:12.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:12.735+0000 7f0b38258700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0b300f6510 0x7f0b30106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:12.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:12.735+0000 7f0b38258700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b30106c20 con 0x7f0b300f6510 2024-01-21T20:04:12.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:12.735+0000 7f0b35ff4700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0b300f6510 0x7f0b30106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:12.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:12.735+0000 7f0b35ff4700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0b300f6510 0x7f0b30106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:58362/0 (socket says 172.21.15.146:58362) 2024-01-21T20:04:12.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:12.735+0000 7f0b35ff4700 1 -- 172.21.15.146:0/2357241667 learned_addr learned my addr 172.21.15.146:0/2357241667 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:04:12.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:12.736+0000 7f0b35ff4700 1 -- 172.21.15.146:0/2357241667 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b20005040 con 0x7f0b300f6510 2024-01-21T20:04:12.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:12.736+0000 7f0b35ff4700 1 --2- 172.21.15.146:0/2357241667 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0b300f6510 0x7f0b30106730 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f0b20013510 tx=0x7f0b2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:12.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:12.736+0000 7f0b267fc700 1 -- 172.21.15.146:0/2357241667 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b20009320 con 0x7f0b300f6510 2024-01-21T20:04:12.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:12.736+0000 7f0b38258700 1 -- 172.21.15.146:0/2357241667 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b30106e20 con 0x7f0b300f6510 2024-01-21T20:04:12.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:12.736+0000 7f0b267fc700 1 -- 172.21.15.146:0/2357241667 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0b20013f00 con 0x7f0b300f6510 2024-01-21T20:04:12.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:12.736+0000 7f0b267fc700 1 -- 172.21.15.146:0/2357241667 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b20014d90 con 0x7f0b300f6510 2024-01-21T20:04:12.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:12.736+0000 7f0b38258700 1 -- 172.21.15.146:0/2357241667 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b301072b0 con 0x7f0b300f6510 2024-01-21T20:04:12.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:12.737+0000 7f0b267fc700 1 -- 172.21.15.146:0/2357241667 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0b200095f0 con 0x7f0b300f6510 2024-01-21T20:04:12.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:12.737+0000 7f0b38258700 1 -- 172.21.15.146:0/2357241667 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b30045ad0 con 0x7f0b300f6510 2024-01-21T20:04:12.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:12.737+0000 7f0b267fc700 1 --2- 172.21.15.146:0/2357241667 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0b1c02ff60 0x7f0b1c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:12.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:12.737+0000 7f0b267fc700 1 -- 172.21.15.146:0/2357241667 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0b200410c0 con 0x7f0b300f6510 2024-01-21T20:04:12.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:12.737+0000 7f0b357f3700 1 --2- 172.21.15.146:0/2357241667 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0b1c02ff60 0x7f0b1c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:12.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:12.738+0000 7f0b357f3700 1 --2- 172.21.15.146:0/2357241667 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0b1c02ff60 0x7f0b1c032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f0b2c0096a0 tx=0x7f0b2c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:12.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:12.742+0000 7f0b267fc700 1 -- 172.21.15.146:0/2357241667 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0b20014420 con 0x7f0b300f6510 2024-01-21T20:04:12.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:12 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:04:11.173276+0000 mgr.smithi053.fgsmol (mgr.14164) 152 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:12.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:12.945+0000 7f0b38258700 1 -- 172.21.15.146:0/2357241667 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0b300fbf60 con 0x7f0b300f6510 2024-01-21T20:04:12.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:12.945+0000 7f0b267fc700 1 -- 172.21.15.146:0/2357241667 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0b2001d7f0 con 0x7f0b300f6510 2024-01-21T20:04:12.937 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:04:12.937 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:04:12.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:12.947+0000 7f0b38258700 1 -- 172.21.15.146:0/2357241667 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0b1c02ff60 msgr2=0x7f0b1c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:12.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:12.947+0000 7f0b38258700 1 --2- 172.21.15.146:0/2357241667 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0b1c02ff60 0x7f0b1c032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f0b2c0096a0 tx=0x7f0b2c006b40).stop 2024-01-21T20:04:12.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:12.947+0000 7f0b38258700 1 -- 172.21.15.146:0/2357241667 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0b300f6510 msgr2=0x7f0b30106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:12.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:12.947+0000 7f0b38258700 1 --2- 172.21.15.146:0/2357241667 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0b300f6510 0x7f0b30106730 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f0b20013510 tx=0x7f0b2000b040).stop 2024-01-21T20:04:12.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:12.947+0000 7f0b38258700 1 -- 172.21.15.146:0/2357241667 shutdown_connections 2024-01-21T20:04:12.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:12.947+0000 7f0b38258700 1 --2- 172.21.15.146:0/2357241667 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0b1c02ff60 0x7f0b1c032420 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:12.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:12.947+0000 7f0b38258700 1 --2- 172.21.15.146:0/2357241667 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0b300f6510 0x7f0b30106730 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:12.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:12.947+0000 7f0b38258700 1 -- 172.21.15.146:0/2357241667 >> 172.21.15.146:0/2357241667 conn(0x7f0b300f1c70 msgr2=0x7f0b300f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:12.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:12.947+0000 7f0b38258700 1 -- 172.21.15.146:0/2357241667 shutdown_connections 2024-01-21T20:04:12.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:12.947+0000 7f0b38258700 1 -- 172.21.15.146:0/2357241667 wait complete. 2024-01-21T20:04:12.940 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:04:13.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:13 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:04:12.946950+0000 mon.smithi053 (mon.0) 223 : audit [DBG] from='client.? 172.21.15.146:0/2357241667' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:04:14.345 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:04:14.345 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:04:14.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:14 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:04:13.173654+0000 mgr.smithi053.fgsmol (mgr.14164) 153 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:15.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:15.592+0000 7f2e2d5dd700 1 -- 172.21.15.146:0/3111920760 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e280f47c0 msgr2=0x7f2e280f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:15.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:15.592+0000 7f2e2d5dd700 1 --2- 172.21.15.146:0/3111920760 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e280f47c0 0x7f2e280f4be0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f2e1c009fb0 tx=0x7f2e1c009b30).stop 2024-01-21T20:04:15.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:15.592+0000 7f2e2d5dd700 1 -- 172.21.15.146:0/3111920760 shutdown_connections 2024-01-21T20:04:15.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:15.592+0000 7f2e2d5dd700 1 --2- 172.21.15.146:0/3111920760 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e280f47c0 0x7f2e280f4be0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:15.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:15.592+0000 7f2e2d5dd700 1 -- 172.21.15.146:0/3111920760 >> 172.21.15.146:0/3111920760 conn(0x7f2e280f02e0 msgr2=0x7f2e280f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:15.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:15.592+0000 7f2e2d5dd700 1 -- 172.21.15.146:0/3111920760 shutdown_connections 2024-01-21T20:04:15.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:15.592+0000 7f2e2d5dd700 1 -- 172.21.15.146:0/3111920760 wait complete. 2024-01-21T20:04:15.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:15.593+0000 7f2e2d5dd700 1 Processor -- start 2024-01-21T20:04:15.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:15.593+0000 7f2e2d5dd700 1 -- start start 2024-01-21T20:04:15.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:15.593+0000 7f2e2d5dd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e280f47c0 0x7f2e28106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:15.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:15.593+0000 7f2e2d5dd700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e28106bf0 con 0x7f2e280f47c0 2024-01-21T20:04:15.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:15.594+0000 7f2e26ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e280f47c0 0x7f2e28106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:15.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:15.594+0000 7f2e26ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e280f47c0 0x7f2e28106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:58376/0 (socket says 172.21.15.146:58376) 2024-01-21T20:04:15.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:15.594+0000 7f2e26ffd700 1 -- 172.21.15.146:0/3087971540 learned_addr learned my addr 172.21.15.146:0/3087971540 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:04:15.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:15.594+0000 7f2e26ffd700 1 -- 172.21.15.146:0/3087971540 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e1c005040 con 0x7f2e280f47c0 2024-01-21T20:04:15.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:15.594+0000 7f2e26ffd700 1 --2- 172.21.15.146:0/3087971540 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e280f47c0 0x7f2e28106700 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f2e1c013510 tx=0x7f2e1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:15.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:15.594+0000 7f2e0f7fe700 1 -- 172.21.15.146:0/3087971540 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e1c009320 con 0x7f2e280f47c0 2024-01-21T20:04:15.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:15.594+0000 7f2e2d5dd700 1 -- 172.21.15.146:0/3087971540 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e28106df0 con 0x7f2e280f47c0 2024-01-21T20:04:15.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:15.594+0000 7f2e0f7fe700 1 -- 172.21.15.146:0/3087971540 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2e1c013f00 con 0x7f2e280f47c0 2024-01-21T20:04:15.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:15.594+0000 7f2e0f7fe700 1 -- 172.21.15.146:0/3087971540 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e1c014d90 con 0x7f2e280f47c0 2024-01-21T20:04:15.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:15.594+0000 7f2e2d5dd700 1 -- 172.21.15.146:0/3087971540 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e28107280 con 0x7f2e280f47c0 2024-01-21T20:04:15.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:15.595+0000 7f2e0f7fe700 1 -- 172.21.15.146:0/3087971540 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2e1c014420 con 0x7f2e280f47c0 2024-01-21T20:04:15.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:15.595+0000 7f2e2d5dd700 1 -- 172.21.15.146:0/3087971540 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e28045ad0 con 0x7f2e280f47c0 2024-01-21T20:04:15.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:15.595+0000 7f2e0f7fe700 1 --2- 172.21.15.146:0/3087971540 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f2e1002ff60 0x7f2e10032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:15.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:15.595+0000 7f2e0f7fe700 1 -- 172.21.15.146:0/3087971540 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2e1c041000 con 0x7f2e280f47c0 2024-01-21T20:04:15.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:15.595+0000 7f2e267fc700 1 --2- 172.21.15.146:0/3087971540 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f2e1002ff60 0x7f2e10032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:15.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:15.596+0000 7f2e267fc700 1 --2- 172.21.15.146:0/3087971540 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f2e1002ff60 0x7f2e10032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f2e180096a0 tx=0x7f2e18006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:15.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:15.600+0000 7f2e0f7fe700 1 -- 172.21.15.146:0/3087971540 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2e1c0095f0 con 0x7f2e280f47c0 2024-01-21T20:04:15.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:15.804+0000 7f2e2d5dd700 1 -- 172.21.15.146:0/3087971540 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2e28108130 con 0x7f2e280f47c0 2024-01-21T20:04:15.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:15.805+0000 7f2e0f7fe700 1 -- 172.21.15.146:0/3087971540 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2e1c01db70 con 0x7f2e280f47c0 2024-01-21T20:04:15.796 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:04:15.796 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:04:15.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:15.807+0000 7f2e2d5dd700 1 -- 172.21.15.146:0/3087971540 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f2e1002ff60 msgr2=0x7f2e10032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:15.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:15.807+0000 7f2e2d5dd700 1 --2- 172.21.15.146:0/3087971540 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f2e1002ff60 0x7f2e10032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f2e180096a0 tx=0x7f2e18006b40).stop 2024-01-21T20:04:15.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:15.807+0000 7f2e2d5dd700 1 -- 172.21.15.146:0/3087971540 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e280f47c0 msgr2=0x7f2e28106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:15.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:15.807+0000 7f2e2d5dd700 1 --2- 172.21.15.146:0/3087971540 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e280f47c0 0x7f2e28106700 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f2e1c013510 tx=0x7f2e1c00b040).stop 2024-01-21T20:04:15.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:15.807+0000 7f2e2d5dd700 1 -- 172.21.15.146:0/3087971540 shutdown_connections 2024-01-21T20:04:15.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:15.807+0000 7f2e2d5dd700 1 --2- 172.21.15.146:0/3087971540 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f2e1002ff60 0x7f2e10032420 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:15.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:15.807+0000 7f2e2d5dd700 1 --2- 172.21.15.146:0/3087971540 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e280f47c0 0x7f2e28106700 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:15.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:15.807+0000 7f2e2d5dd700 1 -- 172.21.15.146:0/3087971540 >> 172.21.15.146:0/3087971540 conn(0x7f2e280f02e0 msgr2=0x7f2e280f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:15.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:15.807+0000 7f2e2d5dd700 1 -- 172.21.15.146:0/3087971540 shutdown_connections 2024-01-21T20:04:15.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:15.807+0000 7f2e2d5dd700 1 -- 172.21.15.146:0/3087971540 wait complete. 2024-01-21T20:04:15.799 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:04:16.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:04:15.173902+0000 mgr.smithi053.fgsmol (mgr.14164) 154 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:16.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:04:15.806536+0000 mon.smithi053 (mon.0) 224 : audit [DBG] from='client.? 172.21.15.146:0/3087971540' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:04:17.209 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:04:17.209 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:04:18.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:18.476+0000 7ff64ca85700 1 -- 172.21.15.146:0/3282317830 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6480601e0 msgr2=0x7ff6480f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:18.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:18.476+0000 7ff64ca85700 1 --2- 172.21.15.146:0/3282317830 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6480601e0 0x7ff6480f8f20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7ff638009fb0 tx=0x7ff638009b30).stop 2024-01-21T20:04:18.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:18.476+0000 7ff64ca85700 1 -- 172.21.15.146:0/3282317830 shutdown_connections 2024-01-21T20:04:18.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:18.476+0000 7ff64ca85700 1 --2- 172.21.15.146:0/3282317830 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6480601e0 0x7ff6480f8f20 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:18.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:18.476+0000 7ff64ca85700 1 -- 172.21.15.146:0/3282317830 >> 172.21.15.146:0/3282317830 conn(0x7ff6480f09a0 msgr2=0x7ff6480f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:18.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:18.476+0000 7ff64ca85700 1 -- 172.21.15.146:0/3282317830 shutdown_connections 2024-01-21T20:04:18.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:18.476+0000 7ff64ca85700 1 -- 172.21.15.146:0/3282317830 wait complete. 2024-01-21T20:04:18.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:18.477+0000 7ff64ca85700 1 Processor -- start 2024-01-21T20:04:18.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:18.477+0000 7ff64ca85700 1 -- start start 2024-01-21T20:04:18.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:18.477+0000 7ff64ca85700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6480601e0 0x7ff6480f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:18.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:18.477+0000 7ff64ca85700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6480f4e20 con 0x7ff6480601e0 2024-01-21T20:04:18.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:18.477+0000 7ff6477fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6480601e0 0x7ff6480f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:18.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:18.477+0000 7ff6477fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6480601e0 0x7ff6480f69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:53706/0 (socket says 172.21.15.146:53706) 2024-01-21T20:04:18.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:18.477+0000 7ff6477fe700 1 -- 172.21.15.146:0/2088302558 learned_addr learned my addr 172.21.15.146:0/2088302558 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:04:18.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:18.478+0000 7ff6477fe700 1 -- 172.21.15.146:0/2088302558 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff638005040 con 0x7ff6480601e0 2024-01-21T20:04:18.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:18.478+0000 7ff6477fe700 1 --2- 172.21.15.146:0/2088302558 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6480601e0 0x7ff6480f69a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7ff638014000 tx=0x7ff63800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:18.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:18.478+0000 7ff62ffff700 1 -- 172.21.15.146:0/2088302558 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6380095f0 con 0x7ff6480601e0 2024-01-21T20:04:18.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:18.478+0000 7ff62ffff700 1 -- 172.21.15.146:0/2088302558 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff638007e70 con 0x7ff6480601e0 2024-01-21T20:04:18.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:18.478+0000 7ff64ca85700 1 -- 172.21.15.146:0/2088302558 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6480f4fc0 con 0x7ff6480601e0 2024-01-21T20:04:18.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:18.478+0000 7ff62ffff700 1 -- 172.21.15.146:0/2088302558 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff638014ea0 con 0x7ff6480601e0 2024-01-21T20:04:18.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:18.478+0000 7ff64ca85700 1 -- 172.21.15.146:0/2088302558 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6480f5450 con 0x7ff6480601e0 2024-01-21T20:04:18.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:18.479+0000 7ff62ffff700 1 -- 172.21.15.146:0/2088302558 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff638025070 con 0x7ff6480601e0 2024-01-21T20:04:18.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:18.479+0000 7ff64ca85700 1 -- 172.21.15.146:0/2088302558 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff648059100 con 0x7ff6480601e0 2024-01-21T20:04:18.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:18.479+0000 7ff62ffff700 1 --2- 172.21.15.146:0/2088302558 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff63002ffb0 0x7ff630032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:18.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:18.479+0000 7ff62ffff700 1 -- 172.21.15.146:0/2088302558 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff638040920 con 0x7ff6480601e0 2024-01-21T20:04:18.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:18.479+0000 7ff646ffd700 1 --2- 172.21.15.146:0/2088302558 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff63002ffb0 0x7ff630032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:18.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:18.480+0000 7ff646ffd700 1 --2- 172.21.15.146:0/2088302558 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff63002ffb0 0x7ff630032470 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7ff63c0096a0 tx=0x7ff63c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:18.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:18.484+0000 7ff62ffff700 1 -- 172.21.15.146:0/2088302558 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff6380079c0 con 0x7ff6480601e0 2024-01-21T20:04:18.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:18.688+0000 7ff64ca85700 1 -- 172.21.15.146:0/2088302558 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff6480fbcd0 con 0x7ff6480601e0 2024-01-21T20:04:18.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:18.689+0000 7ff62ffff700 1 -- 172.21.15.146:0/2088302558 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff63800f300 con 0x7ff6480601e0 2024-01-21T20:04:18.680 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:04:18.680 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:04:18.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:18.691+0000 7ff64ca85700 1 -- 172.21.15.146:0/2088302558 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff63002ffb0 msgr2=0x7ff630032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:18.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:18.691+0000 7ff64ca85700 1 --2- 172.21.15.146:0/2088302558 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff63002ffb0 0x7ff630032470 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7ff63c0096a0 tx=0x7ff63c006b40).stop 2024-01-21T20:04:18.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:18.691+0000 7ff64ca85700 1 -- 172.21.15.146:0/2088302558 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6480601e0 msgr2=0x7ff6480f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:18.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:18.691+0000 7ff64ca85700 1 --2- 172.21.15.146:0/2088302558 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6480601e0 0x7ff6480f69a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7ff638014000 tx=0x7ff63800b040).stop 2024-01-21T20:04:18.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:18.691+0000 7ff64ca85700 1 -- 172.21.15.146:0/2088302558 shutdown_connections 2024-01-21T20:04:18.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:18.691+0000 7ff64ca85700 1 --2- 172.21.15.146:0/2088302558 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff63002ffb0 0x7ff630032470 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:18.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:18.691+0000 7ff64ca85700 1 --2- 172.21.15.146:0/2088302558 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6480601e0 0x7ff6480f69a0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:18.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:18.691+0000 7ff64ca85700 1 -- 172.21.15.146:0/2088302558 >> 172.21.15.146:0/2088302558 conn(0x7ff6480f09a0 msgr2=0x7ff6480f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:18.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:18.691+0000 7ff64ca85700 1 -- 172.21.15.146:0/2088302558 shutdown_connections 2024-01-21T20:04:18.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:18.691+0000 7ff64ca85700 1 -- 172.21.15.146:0/2088302558 wait complete. 2024-01-21T20:04:18.683 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:04:18.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:18 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:04:17.174242+0000 mgr.smithi053.fgsmol (mgr.14164) 155 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:19.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:19 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:04:18.690350+0000 mon.smithi053 (mon.0) 225 : audit [DBG] from='client.? 172.21.15.146:0/2088302558' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:04:20.065 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:04:20.065 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:04:20.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:20 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:04:19.174565+0000 mgr.smithi053.fgsmol (mgr.14164) 156 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:21.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:21.329+0000 7f1532840700 1 -- 172.21.15.146:0/3722951472 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f152c060280 msgr2=0x7f152c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:21.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:21.329+0000 7f1532840700 1 --2- 172.21.15.146:0/3722951472 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f152c060280 0x7f152c0f90f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f151c009fb0 tx=0x7f151c009b30).stop 2024-01-21T20:04:21.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:21.329+0000 7f1532840700 1 -- 172.21.15.146:0/3722951472 shutdown_connections 2024-01-21T20:04:21.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:21.329+0000 7f1532840700 1 --2- 172.21.15.146:0/3722951472 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f152c060280 0x7f152c0f90f0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:21.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:21.329+0000 7f1532840700 1 -- 172.21.15.146:0/3722951472 >> 172.21.15.146:0/3722951472 conn(0x7f152c0f0b70 msgr2=0x7f152c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:21.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:21.329+0000 7f1532840700 1 -- 172.21.15.146:0/3722951472 shutdown_connections 2024-01-21T20:04:21.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:21.329+0000 7f1532840700 1 -- 172.21.15.146:0/3722951472 wait complete. 2024-01-21T20:04:21.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:21.330+0000 7f1532840700 1 Processor -- start 2024-01-21T20:04:21.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:21.330+0000 7f1532840700 1 -- start start 2024-01-21T20:04:21.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:21.330+0000 7f1532840700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f152c060280 0x7f152c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:21.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:21.330+0000 7f1532840700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f152c106bd0 con 0x7f152c060280 2024-01-21T20:04:21.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:21.330+0000 7f152bfff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f152c060280 0x7f152c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:21.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:21.330+0000 7f152bfff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f152c060280 0x7f152c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:53736/0 (socket says 172.21.15.146:53736) 2024-01-21T20:04:21.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:21.330+0000 7f152bfff700 1 -- 172.21.15.146:0/537337474 learned_addr learned my addr 172.21.15.146:0/537337474 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:04:21.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:21.331+0000 7f152bfff700 1 -- 172.21.15.146:0/537337474 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f151c005040 con 0x7f152c060280 2024-01-21T20:04:21.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:21.331+0000 7f152bfff700 1 --2- 172.21.15.146:0/537337474 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f152c060280 0x7f152c1066e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f151c013510 tx=0x7f151c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:21.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:21.331+0000 7f1528ff9700 1 -- 172.21.15.146:0/537337474 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f151c009320 con 0x7f152c060280 2024-01-21T20:04:21.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:21.331+0000 7f1528ff9700 1 -- 172.21.15.146:0/537337474 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f151c013f00 con 0x7f152c060280 2024-01-21T20:04:21.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:21.331+0000 7f1532840700 1 -- 172.21.15.146:0/537337474 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f152c106dd0 con 0x7f152c060280 2024-01-21T20:04:21.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:21.331+0000 7f1528ff9700 1 -- 172.21.15.146:0/537337474 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f151c014dc0 con 0x7f152c060280 2024-01-21T20:04:21.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:21.331+0000 7f1532840700 1 -- 172.21.15.146:0/537337474 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f152c107260 con 0x7f152c060280 2024-01-21T20:04:21.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:21.332+0000 7f1528ff9700 1 -- 172.21.15.146:0/537337474 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f151c01d3d0 con 0x7f152c060280 2024-01-21T20:04:21.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:21.332+0000 7f1532840700 1 -- 172.21.15.146:0/537337474 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f152c045ad0 con 0x7f152c060280 2024-01-21T20:04:21.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:21.332+0000 7f1528ff9700 1 --2- 172.21.15.146:0/537337474 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f151402ff60 0x7f1514032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:21.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:21.332+0000 7f1528ff9700 1 -- 172.21.15.146:0/537337474 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f151c041080 con 0x7f152c060280 2024-01-21T20:04:21.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:21.332+0000 7f152b7fe700 1 --2- 172.21.15.146:0/537337474 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f151402ff60 0x7f1514032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:21.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:21.333+0000 7f152b7fe700 1 --2- 172.21.15.146:0/537337474 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f151402ff60 0x7f1514032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f15200096a0 tx=0x7f1520006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:21.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:21.337+0000 7f1528ff9700 1 -- 172.21.15.146:0/537337474 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f151c012510 con 0x7f152c060280 2024-01-21T20:04:21.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:21.540+0000 7f1532840700 1 -- 172.21.15.146:0/537337474 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f152c108110 con 0x7f152c060280 2024-01-21T20:04:21.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:21.540+0000 7f1528ff9700 1 -- 172.21.15.146:0/537337474 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f151c012330 con 0x7f152c060280 2024-01-21T20:04:21.532 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:04:21.532 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:04:21.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:21.542+0000 7f1532840700 1 -- 172.21.15.146:0/537337474 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f151402ff60 msgr2=0x7f1514032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:21.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:21.542+0000 7f1532840700 1 --2- 172.21.15.146:0/537337474 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f151402ff60 0x7f1514032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f15200096a0 tx=0x7f1520006b40).stop 2024-01-21T20:04:21.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:21.542+0000 7f1532840700 1 -- 172.21.15.146:0/537337474 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f152c060280 msgr2=0x7f152c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:21.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:21.542+0000 7f1532840700 1 --2- 172.21.15.146:0/537337474 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f152c060280 0x7f152c1066e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f151c013510 tx=0x7f151c00b040).stop 2024-01-21T20:04:21.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:21.542+0000 7f1532840700 1 -- 172.21.15.146:0/537337474 shutdown_connections 2024-01-21T20:04:21.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:21.542+0000 7f1532840700 1 --2- 172.21.15.146:0/537337474 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f151402ff60 0x7f1514032420 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:21.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:21.542+0000 7f1532840700 1 --2- 172.21.15.146:0/537337474 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f152c060280 0x7f152c1066e0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:21.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:21.542+0000 7f1532840700 1 -- 172.21.15.146:0/537337474 >> 172.21.15.146:0/537337474 conn(0x7f152c0f0b70 msgr2=0x7f152c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:21.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:21.542+0000 7f1532840700 1 -- 172.21.15.146:0/537337474 shutdown_connections 2024-01-21T20:04:21.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:21.542+0000 7f1532840700 1 -- 172.21.15.146:0/537337474 wait complete. 2024-01-21T20:04:21.534 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:04:21.684 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:21 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:04:21.394506+0000 mon.smithi053 (mon.0) 226 : audit [INF] from='mgr.14164 172.21.15.53:0/2331671680' entity='mgr.smithi053.fgsmol' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi053", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-21T20:04:21.684 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:21 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:04:21.396905+0000 mon.smithi053 (mon.0) 227 : audit [INF] from='mgr.14164 172.21.15.53:0/2331671680' entity='mgr.smithi053.fgsmol' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi053", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-01-21T20:04:21.684 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:21 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:04:21.397739+0000 mon.smithi053 (mon.0) 228 : audit [DBG] from='mgr.14164 172.21.15.53:0/2331671680' entity='mgr.smithi053.fgsmol' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T20:04:22.553 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:04:21.174890+0000 mgr.smithi053.fgsmol (mgr.14164) 157 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:22.553 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cephadm 2024-01-21T20:04:21.393294+0000 mgr.smithi053.fgsmol (mgr.14164) 158 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi053 ... 2024-01-21T20:04:22.553 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-01-21T20:04:22.554 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-01-21T20:04:22.554 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: stat: stderr time="2024-01-21T19:59:57Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T20:04:22.554 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: stat: stderr time="2024-01-21T20:00:37Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T20:04:22.554 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: stat: stderr time="2024-01-21T20:01:14Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T20:04:22.554 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-21T20:04:22.554 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-01-21T20:04:22.554 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-01-21T20:04:22.554 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: stat: stderr time="2024-01-21T20:02:28Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T20:04:22.554 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: stat: stderr time="2024-01-21T20:03:06Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T20:04:22.554 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: stat: stderr time="2024-01-21T20:03:43Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T20:04:22.554 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-21T20:04:22.554 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: Traceback (most recent call last): 2024-01-21T20:04:22.554 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: File "/var/lib/ceph/57f6722c-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T20:04:22.555 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: main() 2024-01-21T20:04:22.555 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: File "/var/lib/ceph/57f6722c-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T20:04:22.555 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: r = ctx.func(ctx) 2024-01-21T20:04:22.555 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: File "/var/lib/ceph/57f6722c-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T20:04:22.555 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: return func(ctx) 2024-01-21T20:04:22.555 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: File "/var/lib/ceph/57f6722c-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-01-21T20:04:22.555 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-01-21T20:04:22.555 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: File "/var/lib/ceph/57f6722c-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-01-21T20:04:22.555 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-01-21T20:04:22.555 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: File "/var/lib/ceph/57f6722c-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-01-21T20:04:22.555 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: raise RuntimeError('uid/gid not found') 2024-01-21T20:04:22.555 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: RuntimeError: uid/gid not found 2024-01-21T20:04:22.555 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: Traceback (most recent call last): 2024-01-21T20:04:22.555 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-01-21T20:04:22.556 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: yield (conn, connr) 2024-01-21T20:04:22.556 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-01-21T20:04:22.556 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: code, '\n'.join(err))) 2024-01-21T20:04:22.556 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi053 ... 2024-01-21T20:04:22.556 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-01-21T20:04:22.556 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-01-21T20:04:22.556 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: stat: stderr time="2024-01-21T19:59:57Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T20:04:22.556 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: stat: stderr time="2024-01-21T20:00:37Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T20:04:22.556 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: stat: stderr time="2024-01-21T20:01:14Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T20:04:22.556 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-21T20:04:22.556 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-01-21T20:04:22.556 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-01-21T20:04:22.556 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: stat: stderr time="2024-01-21T20:02:28Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T20:04:22.556 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: stat: stderr time="2024-01-21T20:03:06Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T20:04:22.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: stat: stderr time="2024-01-21T20:03:43Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T20:04:22.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-21T20:04:22.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: Traceback (most recent call last): 2024-01-21T20:04:22.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: File "/var/lib/ceph/57f6722c-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T20:04:22.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: main() 2024-01-21T20:04:22.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: File "/var/lib/ceph/57f6722c-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T20:04:22.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: r = ctx.func(ctx) 2024-01-21T20:04:22.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: File "/var/lib/ceph/57f6722c-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T20:04:22.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: return func(ctx) 2024-01-21T20:04:22.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: File "/var/lib/ceph/57f6722c-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-01-21T20:04:22.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-01-21T20:04:22.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: File "/var/lib/ceph/57f6722c-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-01-21T20:04:22.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-01-21T20:04:22.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: File "/var/lib/ceph/57f6722c-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-01-21T20:04:22.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: raise RuntimeError('uid/gid not found') 2024-01-21T20:04:22.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: RuntimeError: uid/gid not found 2024-01-21T20:04:22.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cephadm 2024-01-21T20:04:21.398408+0000 mgr.smithi053.fgsmol (mgr.14164) 159 : cephadm [INF] Deploying daemon crash.smithi053 on smithi053 2024-01-21T20:04:22.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:04:21.541871+0000 mon.smithi053 (mon.0) 229 : audit [DBG] from='client.? 172.21.15.146:0/537337474' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:04:22.905 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:04:22.906 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:04:24.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:24.134+0000 7f7708b87700 1 -- 172.21.15.146:0/174718839 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77040f51a0 msgr2=0x7f77040f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:24.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:24.134+0000 7f7708b87700 1 --2- 172.21.15.146:0/174718839 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77040f51a0 0x7f77040f55c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f76f4009fb0 tx=0x7f76f4009b30).stop 2024-01-21T20:04:24.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:24.134+0000 7f7708b87700 1 -- 172.21.15.146:0/174718839 shutdown_connections 2024-01-21T20:04:24.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:24.134+0000 7f7708b87700 1 --2- 172.21.15.146:0/174718839 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77040f51a0 0x7f77040f55c0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:24.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:24.134+0000 7f7708b87700 1 -- 172.21.15.146:0/174718839 >> 172.21.15.146:0/174718839 conn(0x7f77040f09a0 msgr2=0x7f77040f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:24.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:24.134+0000 7f7708b87700 1 -- 172.21.15.146:0/174718839 shutdown_connections 2024-01-21T20:04:24.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:24.134+0000 7f7708b87700 1 -- 172.21.15.146:0/174718839 wait complete. 2024-01-21T20:04:24.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:24.134+0000 7f7708b87700 1 Processor -- start 2024-01-21T20:04:24.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:24.134+0000 7f7708b87700 1 -- start start 2024-01-21T20:04:24.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:24.135+0000 7f7708b87700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77040f51a0 0x7f77041064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:24.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:24.135+0000 7f7708b87700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77041069d0 con 0x7f77040f51a0 2024-01-21T20:04:24.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:24.135+0000 7f77037fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77040f51a0 0x7f77041064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:24.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:24.135+0000 7f77037fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77040f51a0 0x7f77041064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:53750/0 (socket says 172.21.15.146:53750) 2024-01-21T20:04:24.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:24.135+0000 7f77037fe700 1 -- 172.21.15.146:0/2271313205 learned_addr learned my addr 172.21.15.146:0/2271313205 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:04:24.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:24.135+0000 7f77037fe700 1 -- 172.21.15.146:0/2271313205 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76f4005040 con 0x7f77040f51a0 2024-01-21T20:04:24.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:24.135+0000 7f77037fe700 1 --2- 172.21.15.146:0/2271313205 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77040f51a0 0x7f77041064e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f76f4013510 tx=0x7f76f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:24.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:24.135+0000 7f76ebfff700 1 -- 172.21.15.146:0/2271313205 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76f4009320 con 0x7f77040f51a0 2024-01-21T20:04:24.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:24.135+0000 7f7708b87700 1 -- 172.21.15.146:0/2271313205 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7704106bd0 con 0x7f77040f51a0 2024-01-21T20:04:24.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:24.136+0000 7f76ebfff700 1 -- 172.21.15.146:0/2271313205 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f76f4013f00 con 0x7f77040f51a0 2024-01-21T20:04:24.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:24.136+0000 7f76ebfff700 1 -- 172.21.15.146:0/2271313205 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76f4014dc0 con 0x7f77040f51a0 2024-01-21T20:04:24.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:24.136+0000 7f7708b87700 1 -- 172.21.15.146:0/2271313205 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7704107060 con 0x7f77040f51a0 2024-01-21T20:04:24.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:24.136+0000 7f76ebfff700 1 -- 172.21.15.146:0/2271313205 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f76f40095f0 con 0x7f77040f51a0 2024-01-21T20:04:24.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:24.136+0000 7f7708b87700 1 -- 172.21.15.146:0/2271313205 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7704045ad0 con 0x7f77040f51a0 2024-01-21T20:04:24.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:24.136+0000 7f76ebfff700 1 --2- 172.21.15.146:0/2271313205 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f76ec02ff60 0x7f76ec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:24.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:24.137+0000 7f76ebfff700 1 -- 172.21.15.146:0/2271313205 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f76f40410c0 con 0x7f77040f51a0 2024-01-21T20:04:24.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:24.137+0000 7f7702ffd700 1 --2- 172.21.15.146:0/2271313205 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f76ec02ff60 0x7f76ec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:24.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:24.137+0000 7f7702ffd700 1 --2- 172.21.15.146:0/2271313205 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f76ec02ff60 0x7f76ec032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f76f80096a0 tx=0x7f76f8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:24.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:24.142+0000 7f76ebfff700 1 -- 172.21.15.146:0/2271313205 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f76f401c790 con 0x7f77040f51a0 2024-01-21T20:04:24.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:24.347+0000 7f7708b87700 1 -- 172.21.15.146:0/2271313205 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7704107f10 con 0x7f77040f51a0 2024-01-21T20:04:24.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:24.348+0000 7f76ebfff700 1 -- 172.21.15.146:0/2271313205 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f76f4010230 con 0x7f77040f51a0 2024-01-21T20:04:24.340 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:04:24.340 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:04:24.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:24.349+0000 7f7708b87700 1 -- 172.21.15.146:0/2271313205 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f76ec02ff60 msgr2=0x7f76ec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:24.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:24.349+0000 7f7708b87700 1 --2- 172.21.15.146:0/2271313205 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f76ec02ff60 0x7f76ec032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f76f80096a0 tx=0x7f76f8006b40).stop 2024-01-21T20:04:24.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:24.349+0000 7f7708b87700 1 -- 172.21.15.146:0/2271313205 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77040f51a0 msgr2=0x7f77041064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:24.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:24.349+0000 7f7708b87700 1 --2- 172.21.15.146:0/2271313205 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77040f51a0 0x7f77041064e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f76f4013510 tx=0x7f76f400b040).stop 2024-01-21T20:04:24.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:24.349+0000 7f7708b87700 1 -- 172.21.15.146:0/2271313205 shutdown_connections 2024-01-21T20:04:24.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:24.349+0000 7f7708b87700 1 --2- 172.21.15.146:0/2271313205 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f76ec02ff60 0x7f76ec032420 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:24.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:24.349+0000 7f7708b87700 1 --2- 172.21.15.146:0/2271313205 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77040f51a0 0x7f77041064e0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:24.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:24.349+0000 7f7708b87700 1 -- 172.21.15.146:0/2271313205 >> 172.21.15.146:0/2271313205 conn(0x7f77040f09a0 msgr2=0x7f770405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:24.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:24.349+0000 7f7708b87700 1 -- 172.21.15.146:0/2271313205 shutdown_connections 2024-01-21T20:04:24.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:24.349+0000 7f7708b87700 1 -- 172.21.15.146:0/2271313205 wait complete. 2024-01-21T20:04:24.344 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:04:24.433 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:24 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:04:23.047219+0000 mon.smithi053 (mon.0) 230 : audit [INF] from='mgr.14164 172.21.15.53:0/2331671680' entity='mgr.smithi053.fgsmol' 2024-01-21T20:04:24.434 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:24 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:04:23.121433+0000 mon.smithi053 (mon.0) 231 : audit [INF] from='mgr.14164 172.21.15.53:0/2331671680' entity='mgr.smithi053.fgsmol' 2024-01-21T20:04:24.434 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:24 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:04:23.123546+0000 mon.smithi053 (mon.0) 232 : audit [INF] from='mgr.14164 172.21.15.53:0/2331671680' entity='mgr.smithi053.fgsmol' 2024-01-21T20:04:24.434 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:24 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:04:23.124019+0000 mon.smithi053 (mon.0) 233 : audit [INF] from='mgr.14164 172.21.15.53:0/2331671680' entity='mgr.smithi053.fgsmol' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-01-21T20:04:24.434 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:24 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:04:23.124511+0000 mgr.smithi053.fgsmol (mgr.14164) 160 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-01-21T20:04:24.434 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:24 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:04:23.126820+0000 mon.smithi053 (mon.0) 234 : audit [INF] from='mgr.14164 172.21.15.53:0/2331671680' entity='mgr.smithi053.fgsmol' 2024-01-21T20:04:24.435 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:24 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cephadm 2024-01-21T20:04:23.130151+0000 mgr.smithi053.fgsmol (mgr.14164) 161 : cephadm [INF] Deploying daemon grafana.smithi053 on smithi053 2024-01-21T20:04:24.435 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:24 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:04:23.175177+0000 mgr.smithi053.fgsmol (mgr.14164) 162 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:25.433 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:25 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:04:24.349021+0000 mon.smithi053 (mon.0) 235 : audit [DBG] from='client.? 172.21.15.146:0/2271313205' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:04:25.722 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:04:25.723 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:04:26.433 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:26 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:04:25.175506+0000 mgr.smithi053.fgsmol (mgr.14164) 163 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:26.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:26.855+0000 7f143441e700 1 -- 172.21.15.146:0/3337246472 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f142c060200 msgr2=0x7f142c0f9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:26.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:26.855+0000 7f143441e700 1 --2- 172.21.15.146:0/3337246472 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f142c060200 0x7f142c0f9170 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f1420009fb0 tx=0x7f1420009b30).stop 2024-01-21T20:04:26.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:26.855+0000 7f143441e700 1 -- 172.21.15.146:0/3337246472 shutdown_connections 2024-01-21T20:04:26.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:26.855+0000 7f143441e700 1 --2- 172.21.15.146:0/3337246472 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f142c060200 0x7f142c0f9170 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:26.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:26.855+0000 7f143441e700 1 -- 172.21.15.146:0/3337246472 >> 172.21.15.146:0/3337246472 conn(0x7f142c0f0b70 msgr2=0x7f142c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:26.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:26.855+0000 7f143441e700 1 -- 172.21.15.146:0/3337246472 shutdown_connections 2024-01-21T20:04:26.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:26.855+0000 7f143441e700 1 -- 172.21.15.146:0/3337246472 wait complete. 2024-01-21T20:04:26.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:26.856+0000 7f143441e700 1 Processor -- start 2024-01-21T20:04:26.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:26.856+0000 7f143441e700 1 -- start start 2024-01-21T20:04:26.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:26.856+0000 7f143441e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f142c060200 0x7f142c1066a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:26.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:26.856+0000 7f143441e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f142c106b90 con 0x7f142c060200 2024-01-21T20:04:26.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:26.856+0000 7f14321ba700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f142c060200 0x7f142c1066a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:26.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:26.856+0000 7f14321ba700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f142c060200 0x7f142c1066a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:53768/0 (socket says 172.21.15.146:53768) 2024-01-21T20:04:26.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:26.856+0000 7f14321ba700 1 -- 172.21.15.146:0/828323328 learned_addr learned my addr 172.21.15.146:0/828323328 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:04:26.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:26.857+0000 7f14321ba700 1 -- 172.21.15.146:0/828323328 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1420005040 con 0x7f142c060200 2024-01-21T20:04:26.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:26.857+0000 7f14321ba700 1 --2- 172.21.15.146:0/828323328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f142c060200 0x7f142c1066a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f1420013510 tx=0x7f142000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:26.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:26.857+0000 7f141effd700 1 -- 172.21.15.146:0/828323328 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1420009320 con 0x7f142c060200 2024-01-21T20:04:26.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:26.857+0000 7f143441e700 1 -- 172.21.15.146:0/828323328 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f142c106d90 con 0x7f142c060200 2024-01-21T20:04:26.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:26.857+0000 7f141effd700 1 -- 172.21.15.146:0/828323328 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1420013f00 con 0x7f142c060200 2024-01-21T20:04:26.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:26.857+0000 7f141effd700 1 -- 172.21.15.146:0/828323328 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1420014dc0 con 0x7f142c060200 2024-01-21T20:04:26.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:26.857+0000 7f143441e700 1 -- 172.21.15.146:0/828323328 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f142c107220 con 0x7f142c060200 2024-01-21T20:04:26.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:26.858+0000 7f143441e700 1 -- 172.21.15.146:0/828323328 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f142c045ad0 con 0x7f142c060200 2024-01-21T20:04:26.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:26.858+0000 7f141effd700 1 -- 172.21.15.146:0/828323328 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f14200095f0 con 0x7f142c060200 2024-01-21T20:04:26.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:26.858+0000 7f141effd700 1 --2- 172.21.15.146:0/828323328 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f141802ff60 0x7f1418032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:26.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:26.858+0000 7f141effd700 1 -- 172.21.15.146:0/828323328 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1420041020 con 0x7f142c060200 2024-01-21T20:04:26.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:26.858+0000 7f14319b9700 1 --2- 172.21.15.146:0/828323328 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f141802ff60 0x7f1418032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:26.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:26.859+0000 7f14319b9700 1 --2- 172.21.15.146:0/828323328 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f141802ff60 0x7f1418032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f14280096a0 tx=0x7f1428006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:26.854 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:26.863+0000 7f141effd700 1 -- 172.21.15.146:0/828323328 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1420014420 con 0x7f142c060200 2024-01-21T20:04:27.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:27.069+0000 7f143441e700 1 -- 172.21.15.146:0/828323328 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f142c108070 con 0x7f142c060200 2024-01-21T20:04:27.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:27.069+0000 7f141effd700 1 -- 172.21.15.146:0/828323328 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f142001c920 con 0x7f142c060200 2024-01-21T20:04:27.060 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:04:27.061 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:04:27.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:27.071+0000 7f143441e700 1 -- 172.21.15.146:0/828323328 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f141802ff60 msgr2=0x7f1418032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:27.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:27.071+0000 7f143441e700 1 --2- 172.21.15.146:0/828323328 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f141802ff60 0x7f1418032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f14280096a0 tx=0x7f1428006b40).stop 2024-01-21T20:04:27.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:27.071+0000 7f143441e700 1 -- 172.21.15.146:0/828323328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f142c060200 msgr2=0x7f142c1066a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:27.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:27.071+0000 7f143441e700 1 --2- 172.21.15.146:0/828323328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f142c060200 0x7f142c1066a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f1420013510 tx=0x7f142000b040).stop 2024-01-21T20:04:27.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:27.071+0000 7f143441e700 1 -- 172.21.15.146:0/828323328 shutdown_connections 2024-01-21T20:04:27.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:27.071+0000 7f143441e700 1 --2- 172.21.15.146:0/828323328 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f141802ff60 0x7f1418032420 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:27.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:27.071+0000 7f143441e700 1 --2- 172.21.15.146:0/828323328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f142c060200 0x7f142c1066a0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:27.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:27.071+0000 7f143441e700 1 -- 172.21.15.146:0/828323328 >> 172.21.15.146:0/828323328 conn(0x7f142c0f0b70 msgr2=0x7f142c0f1620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:27.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:27.071+0000 7f143441e700 1 -- 172.21.15.146:0/828323328 shutdown_connections 2024-01-21T20:04:27.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:27.071+0000 7f143441e700 1 -- 172.21.15.146:0/828323328 wait complete. 2024-01-21T20:04:27.063 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:04:27.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:27 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:04:27.070768+0000 mon.smithi053 (mon.0) 236 : audit [DBG] from='client.? 172.21.15.146:0/828323328' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:04:28.466 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:04:28.466 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:04:28.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:28 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:04:27.175830+0000 mgr.smithi053.fgsmol (mgr.14164) 164 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:29.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:29.657+0000 7f7fae380700 1 -- 172.21.15.146:0/2360202338 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fa80f6510 msgr2=0x7f7fa80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:29.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:29.657+0000 7f7fae380700 1 --2- 172.21.15.146:0/2360202338 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fa80f6510 0x7f7fa80f6930 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f7f98009fb0 tx=0x7f7f98009b30).stop 2024-01-21T20:04:29.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:29.657+0000 7f7fae380700 1 -- 172.21.15.146:0/2360202338 shutdown_connections 2024-01-21T20:04:29.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:29.657+0000 7f7fae380700 1 --2- 172.21.15.146:0/2360202338 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fa80f6510 0x7f7fa80f6930 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:29.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:29.657+0000 7f7fae380700 1 -- 172.21.15.146:0/2360202338 >> 172.21.15.146:0/2360202338 conn(0x7f7fa80f1c70 msgr2=0x7f7fa80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:29.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:29.657+0000 7f7fae380700 1 -- 172.21.15.146:0/2360202338 shutdown_connections 2024-01-21T20:04:29.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:29.657+0000 7f7fae380700 1 -- 172.21.15.146:0/2360202338 wait complete. 2024-01-21T20:04:29.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:29.658+0000 7f7fae380700 1 Processor -- start 2024-01-21T20:04:29.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:29.658+0000 7f7fae380700 1 -- start start 2024-01-21T20:04:29.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:29.658+0000 7f7fae380700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fa80f6510 0x7f7fa8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:29.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:29.658+0000 7f7fae380700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa8106c20 con 0x7f7fa80f6510 2024-01-21T20:04:29.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:29.658+0000 7f7fa7fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fa80f6510 0x7f7fa8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:29.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:29.658+0000 7f7fa7fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fa80f6510 0x7f7fa8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:59522/0 (socket says 172.21.15.146:59522) 2024-01-21T20:04:29.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:29.658+0000 7f7fa7fff700 1 -- 172.21.15.146:0/1681790546 learned_addr learned my addr 172.21.15.146:0/1681790546 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:04:29.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:29.659+0000 7f7fa7fff700 1 -- 172.21.15.146:0/1681790546 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f98005040 con 0x7f7fa80f6510 2024-01-21T20:04:29.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:29.659+0000 7f7fa7fff700 1 --2- 172.21.15.146:0/1681790546 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fa80f6510 0x7f7fa8106730 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f7f98013510 tx=0x7f7f9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:29.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:29.659+0000 7f7fa4ff9700 1 -- 172.21.15.146:0/1681790546 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f98009320 con 0x7f7fa80f6510 2024-01-21T20:04:29.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:29.659+0000 7f7fae380700 1 -- 172.21.15.146:0/1681790546 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fa8106e20 con 0x7f7fa80f6510 2024-01-21T20:04:29.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:29.659+0000 7f7fa4ff9700 1 -- 172.21.15.146:0/1681790546 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7f98013f00 con 0x7f7fa80f6510 2024-01-21T20:04:29.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:29.659+0000 7f7fa4ff9700 1 -- 172.21.15.146:0/1681790546 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f98014d90 con 0x7f7fa80f6510 2024-01-21T20:04:29.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:29.659+0000 7f7fae380700 1 -- 172.21.15.146:0/1681790546 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fa81072b0 con 0x7f7fa80f6510 2024-01-21T20:04:29.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:29.660+0000 7f7fa4ff9700 1 -- 172.21.15.146:0/1681790546 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7f980095f0 con 0x7f7fa80f6510 2024-01-21T20:04:29.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:29.660+0000 7f7fae380700 1 -- 172.21.15.146:0/1681790546 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fa8045ad0 con 0x7f7fa80f6510 2024-01-21T20:04:29.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:29.660+0000 7f7fa4ff9700 1 --2- 172.21.15.146:0/1681790546 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f7f9402ff60 0x7f7f94032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:29.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:29.660+0000 7f7fa4ff9700 1 -- 172.21.15.146:0/1681790546 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7f98041000 con 0x7f7fa80f6510 2024-01-21T20:04:29.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:29.660+0000 7f7fa77fe700 1 --2- 172.21.15.146:0/1681790546 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f7f9402ff60 0x7f7f94032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:29.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:29.661+0000 7f7fa77fe700 1 --2- 172.21.15.146:0/1681790546 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f7f9402ff60 0x7f7f94032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f7f900096a0 tx=0x7f7f90006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:29.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:29.665+0000 7f7fa4ff9700 1 -- 172.21.15.146:0/1681790546 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7f98014420 con 0x7f7fa80f6510 2024-01-21T20:04:29.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:29.874+0000 7f7fae380700 1 -- 172.21.15.146:0/1681790546 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7fa80fbf60 con 0x7f7fa80f6510 2024-01-21T20:04:29.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:29.875+0000 7f7fa4ff9700 1 -- 172.21.15.146:0/1681790546 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7f9801d7f0 con 0x7f7fa80f6510 2024-01-21T20:04:29.865 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:04:29.866 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:04:29.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:29.876+0000 7f7fae380700 1 -- 172.21.15.146:0/1681790546 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f7f9402ff60 msgr2=0x7f7f94032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:29.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:29.876+0000 7f7fae380700 1 --2- 172.21.15.146:0/1681790546 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f7f9402ff60 0x7f7f94032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f7f900096a0 tx=0x7f7f90006b40).stop 2024-01-21T20:04:29.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:29.876+0000 7f7fae380700 1 -- 172.21.15.146:0/1681790546 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fa80f6510 msgr2=0x7f7fa8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:29.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:29.876+0000 7f7fae380700 1 --2- 172.21.15.146:0/1681790546 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fa80f6510 0x7f7fa8106730 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f7f98013510 tx=0x7f7f9800b040).stop 2024-01-21T20:04:29.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:29.876+0000 7f7fae380700 1 -- 172.21.15.146:0/1681790546 shutdown_connections 2024-01-21T20:04:29.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:29.876+0000 7f7fae380700 1 --2- 172.21.15.146:0/1681790546 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f7f9402ff60 0x7f7f94032420 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:29.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:29.876+0000 7f7fae380700 1 --2- 172.21.15.146:0/1681790546 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fa80f6510 0x7f7fa8106730 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:29.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:29.876+0000 7f7fae380700 1 -- 172.21.15.146:0/1681790546 >> 172.21.15.146:0/1681790546 conn(0x7f7fa80f1c70 msgr2=0x7f7fa80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:29.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:29.876+0000 7f7fae380700 1 -- 172.21.15.146:0/1681790546 shutdown_connections 2024-01-21T20:04:29.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:29.876+0000 7f7fae380700 1 -- 172.21.15.146:0/1681790546 wait complete. 2024-01-21T20:04:29.868 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:04:30.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:30 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:04:29.176181+0000 mgr.smithi053.fgsmol (mgr.14164) 165 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:30.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:30 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:04:29.875930+0000 mon.smithi053 (mon.0) 237 : audit [DBG] from='client.? 172.21.15.146:0/1681790546' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:04:31.290 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:04:31.290 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:04:32.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:32.512+0000 7fa1ecace700 1 -- 172.21.15.146:0/829268761 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1e80f51a0 msgr2=0x7fa1e80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:32.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:32.512+0000 7fa1ecace700 1 --2- 172.21.15.146:0/829268761 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1e80f51a0 0x7fa1e80f55c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7fa1e0009960 tx=0x7fa1e0004d10).stop 2024-01-21T20:04:32.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:32.512+0000 7fa1ecace700 1 -- 172.21.15.146:0/829268761 shutdown_connections 2024-01-21T20:04:32.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:32.512+0000 7fa1ecace700 1 --2- 172.21.15.146:0/829268761 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1e80f51a0 0x7fa1e80f55c0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:32.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:32.512+0000 7fa1ecace700 1 -- 172.21.15.146:0/829268761 >> 172.21.15.146:0/829268761 conn(0x7fa1e80f09a0 msgr2=0x7fa1e80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:32.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:32.512+0000 7fa1ecace700 1 -- 172.21.15.146:0/829268761 shutdown_connections 2024-01-21T20:04:32.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:32.512+0000 7fa1ecace700 1 -- 172.21.15.146:0/829268761 wait complete. 2024-01-21T20:04:32.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:32.512+0000 7fa1ecace700 1 Processor -- start 2024-01-21T20:04:32.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:32.513+0000 7fa1ecace700 1 -- start start 2024-01-21T20:04:32.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:32.513+0000 7fa1ecace700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1e80f51a0 0x7fa1e81091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:32.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:32.513+0000 7fa1ecace700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e8060220 con 0x7fa1e80f51a0 2024-01-21T20:04:32.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:32.513+0000 7fa1e77fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1e80f51a0 0x7fa1e81091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:32.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:32.513+0000 7fa1e77fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1e80f51a0 0x7fa1e81091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:59540/0 (socket says 172.21.15.146:59540) 2024-01-21T20:04:32.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:32.513+0000 7fa1e77fe700 1 -- 172.21.15.146:0/3309772905 learned_addr learned my addr 172.21.15.146:0/3309772905 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:04:32.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:32.513+0000 7fa1e77fe700 1 -- 172.21.15.146:0/3309772905 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e0009640 con 0x7fa1e80f51a0 2024-01-21T20:04:32.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:32.513+0000 7fa1e77fe700 1 --2- 172.21.15.146:0/3309772905 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1e80f51a0 0x7fa1e81091d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fa1e0007bd0 tx=0x7fa1e0004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:32.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:32.514+0000 7fa1cffff700 1 -- 172.21.15.146:0/3309772905 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1e0018070 con 0x7fa1e80f51a0 2024-01-21T20:04:32.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:32.514+0000 7fa1ecace700 1 -- 172.21.15.146:0/3309772905 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1e8060420 con 0x7fa1e80f51a0 2024-01-21T20:04:32.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:32.514+0000 7fa1cffff700 1 -- 172.21.15.146:0/3309772905 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa1e000b870 con 0x7fa1e80f51a0 2024-01-21T20:04:32.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:32.514+0000 7fa1cffff700 1 -- 172.21.15.146:0/3309772905 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1e001d420 con 0x7fa1e80f51a0 2024-01-21T20:04:32.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:32.514+0000 7fa1ecace700 1 -- 172.21.15.146:0/3309772905 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1e8109880 con 0x7fa1e80f51a0 2024-01-21T20:04:32.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:32.514+0000 7fa1cffff700 1 -- 172.21.15.146:0/3309772905 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa1e0024060 con 0x7fa1e80f51a0 2024-01-21T20:04:32.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:32.514+0000 7fa1ecace700 1 -- 172.21.15.146:0/3309772905 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1e8059100 con 0x7fa1e80f51a0 2024-01-21T20:04:32.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:32.515+0000 7fa1cffff700 1 --2- 172.21.15.146:0/3309772905 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa1d002ff60 0x7fa1d0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:32.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:32.515+0000 7fa1cffff700 1 -- 172.21.15.146:0/3309772905 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa1e0041250 con 0x7fa1e80f51a0 2024-01-21T20:04:32.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:32.515+0000 7fa1e6ffd700 1 --2- 172.21.15.146:0/3309772905 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa1d002ff60 0x7fa1d0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:32.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:32.515+0000 7fa1e6ffd700 1 --2- 172.21.15.146:0/3309772905 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa1d002ff60 0x7fa1d0032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fa1dc0096a0 tx=0x7fa1dc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:32.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:32.519+0000 7fa1cffff700 1 -- 172.21.15.146:0/3309772905 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa1e0011790 con 0x7fa1e80f51a0 2024-01-21T20:04:32.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:32.725+0000 7fa1ecace700 1 -- 172.21.15.146:0/3309772905 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa1e8045ad0 con 0x7fa1e80f51a0 2024-01-21T20:04:32.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:32.725+0000 7fa1cffff700 1 -- 172.21.15.146:0/3309772905 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa1e0011790 con 0x7fa1e80f51a0 2024-01-21T20:04:32.717 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:04:32.717 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:04:32.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:32.727+0000 7fa1ecace700 1 -- 172.21.15.146:0/3309772905 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa1d002ff60 msgr2=0x7fa1d0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:32.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:32.727+0000 7fa1ecace700 1 --2- 172.21.15.146:0/3309772905 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa1d002ff60 0x7fa1d0032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fa1dc0096a0 tx=0x7fa1dc006b40).stop 2024-01-21T20:04:32.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:32.727+0000 7fa1ecace700 1 -- 172.21.15.146:0/3309772905 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1e80f51a0 msgr2=0x7fa1e81091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:32.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:32.727+0000 7fa1ecace700 1 --2- 172.21.15.146:0/3309772905 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1e80f51a0 0x7fa1e81091d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fa1e0007bd0 tx=0x7fa1e0004d10).stop 2024-01-21T20:04:32.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:32.727+0000 7fa1ecace700 1 -- 172.21.15.146:0/3309772905 shutdown_connections 2024-01-21T20:04:32.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:32.727+0000 7fa1ecace700 1 --2- 172.21.15.146:0/3309772905 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa1d002ff60 0x7fa1d0032420 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:32.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:32.727+0000 7fa1ecace700 1 --2- 172.21.15.146:0/3309772905 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1e80f51a0 0x7fa1e81091d0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:32.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:32.727+0000 7fa1ecace700 1 -- 172.21.15.146:0/3309772905 >> 172.21.15.146:0/3309772905 conn(0x7fa1e80f09a0 msgr2=0x7fa1e80f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:32.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:32.727+0000 7fa1ecace700 1 -- 172.21.15.146:0/3309772905 shutdown_connections 2024-01-21T20:04:32.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:32.727+0000 7fa1ecace700 1 -- 172.21.15.146:0/3309772905 wait complete. 2024-01-21T20:04:32.719 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:04:32.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:32 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:04:31.176483+0000 mgr.smithi053.fgsmol (mgr.14164) 166 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:33.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:33 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:04:32.726928+0000 mon.smithi053 (mon.0) 238 : audit [DBG] from='client.? 172.21.15.146:0/3309772905' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:04:34.165 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:04:34.165 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:04:34.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:34 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:04:33.176712+0000 mgr.smithi053.fgsmol (mgr.14164) 167 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:35.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:35.321+0000 7fee7eb18700 1 -- 172.21.15.146:0/2335845909 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee780f51a0 msgr2=0x7fee780f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:35.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:35.321+0000 7fee7eb18700 1 --2- 172.21.15.146:0/2335845909 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee780f51a0 0x7fee780f55c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7fee68009fb0 tx=0x7fee68009b30).stop 2024-01-21T20:04:35.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:35.321+0000 7fee7eb18700 1 -- 172.21.15.146:0/2335845909 shutdown_connections 2024-01-21T20:04:35.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:35.321+0000 7fee7eb18700 1 --2- 172.21.15.146:0/2335845909 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee780f51a0 0x7fee780f55c0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:35.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:35.321+0000 7fee7eb18700 1 -- 172.21.15.146:0/2335845909 >> 172.21.15.146:0/2335845909 conn(0x7fee780f09a0 msgr2=0x7fee780f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:35.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:35.321+0000 7fee7eb18700 1 -- 172.21.15.146:0/2335845909 shutdown_connections 2024-01-21T20:04:35.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:35.321+0000 7fee7eb18700 1 -- 172.21.15.146:0/2335845909 wait complete. 2024-01-21T20:04:35.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:35.321+0000 7fee7eb18700 1 Processor -- start 2024-01-21T20:04:35.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:35.321+0000 7fee7eb18700 1 -- start start 2024-01-21T20:04:35.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:35.321+0000 7fee7eb18700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee780f51a0 0x7fee781091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:35.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:35.321+0000 7fee7eb18700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee78060220 con 0x7fee780f51a0 2024-01-21T20:04:35.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:35.322+0000 7fee7db16700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee780f51a0 0x7fee781091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:35.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:35.322+0000 7fee7db16700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee780f51a0 0x7fee781091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:59566/0 (socket says 172.21.15.146:59566) 2024-01-21T20:04:35.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:35.322+0000 7fee7db16700 1 -- 172.21.15.146:0/2387662618 learned_addr learned my addr 172.21.15.146:0/2387662618 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:04:35.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:35.322+0000 7fee7db16700 1 -- 172.21.15.146:0/2387662618 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee68005040 con 0x7fee780f51a0 2024-01-21T20:04:35.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:35.322+0000 7fee7db16700 1 --2- 172.21.15.146:0/2387662618 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee780f51a0 0x7fee781091d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7fee68013510 tx=0x7fee6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:35.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:35.322+0000 7fee6e7fc700 1 -- 172.21.15.146:0/2387662618 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee68009320 con 0x7fee780f51a0 2024-01-21T20:04:35.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:35.322+0000 7fee7eb18700 1 -- 172.21.15.146:0/2387662618 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee78060420 con 0x7fee780f51a0 2024-01-21T20:04:35.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:35.322+0000 7fee6e7fc700 1 -- 172.21.15.146:0/2387662618 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fee68013f00 con 0x7fee780f51a0 2024-01-21T20:04:35.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:35.322+0000 7fee6e7fc700 1 -- 172.21.15.146:0/2387662618 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee68014dc0 con 0x7fee780f51a0 2024-01-21T20:04:35.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:35.322+0000 7fee7eb18700 1 -- 172.21.15.146:0/2387662618 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee78109880 con 0x7fee780f51a0 2024-01-21T20:04:35.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:35.323+0000 7fee6e7fc700 1 -- 172.21.15.146:0/2387662618 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fee6801d3d0 con 0x7fee780f51a0 2024-01-21T20:04:35.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:35.323+0000 7fee6e7fc700 1 --2- 172.21.15.146:0/2387662618 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fee6402ffb0 0x7fee64032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:35.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:35.323+0000 7fee6e7fc700 1 -- 172.21.15.146:0/2387662618 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fee680410d0 con 0x7fee780f51a0 2024-01-21T20:04:35.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:35.323+0000 7fee7eb18700 1 -- 172.21.15.146:0/2387662618 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee78059100 con 0x7fee780f51a0 2024-01-21T20:04:35.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:35.323+0000 7fee7d315700 1 --2- 172.21.15.146:0/2387662618 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fee6402ffb0 0x7fee64032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:35.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:35.324+0000 7fee7d315700 1 --2- 172.21.15.146:0/2387662618 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fee6402ffb0 0x7fee64032470 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fee740096a0 tx=0x7fee74006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:35.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:35.328+0000 7fee6e7fc700 1 -- 172.21.15.146:0/2387662618 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fee68012900 con 0x7fee780f51a0 2024-01-21T20:04:35.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:35.533+0000 7fee7eb18700 1 -- 172.21.15.146:0/2387662618 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fee78109fd0 con 0x7fee780f51a0 2024-01-21T20:04:35.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:35.533+0000 7fee6e7fc700 1 -- 172.21.15.146:0/2387662618 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fee6802b020 con 0x7fee780f51a0 2024-01-21T20:04:35.524 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:04:35.524 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:04:35.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:35.535+0000 7fee7eb18700 1 -- 172.21.15.146:0/2387662618 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fee6402ffb0 msgr2=0x7fee64032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:35.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:35.535+0000 7fee7eb18700 1 --2- 172.21.15.146:0/2387662618 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fee6402ffb0 0x7fee64032470 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fee740096a0 tx=0x7fee74006b40).stop 2024-01-21T20:04:35.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:35.535+0000 7fee7eb18700 1 -- 172.21.15.146:0/2387662618 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee780f51a0 msgr2=0x7fee781091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:35.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:35.535+0000 7fee7eb18700 1 --2- 172.21.15.146:0/2387662618 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee780f51a0 0x7fee781091d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7fee68013510 tx=0x7fee6800b040).stop 2024-01-21T20:04:35.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:35.535+0000 7fee7eb18700 1 -- 172.21.15.146:0/2387662618 shutdown_connections 2024-01-21T20:04:35.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:35.535+0000 7fee7eb18700 1 --2- 172.21.15.146:0/2387662618 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fee6402ffb0 0x7fee64032470 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:35.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:35.535+0000 7fee7eb18700 1 --2- 172.21.15.146:0/2387662618 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee780f51a0 0x7fee781091d0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:35.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:35.535+0000 7fee7eb18700 1 -- 172.21.15.146:0/2387662618 >> 172.21.15.146:0/2387662618 conn(0x7fee780f09a0 msgr2=0x7fee780f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:35.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:35.535+0000 7fee7eb18700 1 -- 172.21.15.146:0/2387662618 shutdown_connections 2024-01-21T20:04:35.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:35.535+0000 7fee7eb18700 1 -- 172.21.15.146:0/2387662618 wait complete. 2024-01-21T20:04:35.526 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:04:36.908 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:04:36.908 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:04:36.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:36 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:04:35.176920+0000 mgr.smithi053.fgsmol (mgr.14164) 168 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:36.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:36 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:04:35.533918+0000 mon.smithi053 (mon.0) 239 : audit [DBG] from='client.? 172.21.15.146:0/2387662618' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:04:38.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:38.165+0000 7f2839634700 1 -- 172.21.15.146:0/274795905 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28340f5410 msgr2=0x7f28340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:38.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:38.165+0000 7f2839634700 1 --2- 172.21.15.146:0/274795905 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28340f5410 0x7f28340f5830 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f2828009fb0 tx=0x7f2828009b30).stop 2024-01-21T20:04:38.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:38.165+0000 7f2839634700 1 -- 172.21.15.146:0/274795905 shutdown_connections 2024-01-21T20:04:38.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:38.165+0000 7f2839634700 1 --2- 172.21.15.146:0/274795905 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28340f5410 0x7f28340f5830 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:38.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:38.165+0000 7f2839634700 1 -- 172.21.15.146:0/274795905 >> 172.21.15.146:0/274795905 conn(0x7f28340f0b70 msgr2=0x7f28340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:38.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:38.165+0000 7f2839634700 1 -- 172.21.15.146:0/274795905 shutdown_connections 2024-01-21T20:04:38.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:38.165+0000 7f2839634700 1 -- 172.21.15.146:0/274795905 wait complete. 2024-01-21T20:04:38.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:38.166+0000 7f2839634700 1 Processor -- start 2024-01-21T20:04:38.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:38.166+0000 7f2839634700 1 -- start start 2024-01-21T20:04:38.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:38.166+0000 7f2839634700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28340f5410 0x7f2834109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:38.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:38.166+0000 7f2839634700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2834060240 con 0x7f28340f5410 2024-01-21T20:04:38.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:38.166+0000 7f2832ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28340f5410 0x7f2834109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:38.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:38.166+0000 7f2832ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28340f5410 0x7f2834109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:56812/0 (socket says 172.21.15.146:56812) 2024-01-21T20:04:38.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:38.166+0000 7f2832ffd700 1 -- 172.21.15.146:0/3917418412 learned_addr learned my addr 172.21.15.146:0/3917418412 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:04:38.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:38.167+0000 7f2832ffd700 1 -- 172.21.15.146:0/3917418412 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2828005040 con 0x7f28340f5410 2024-01-21T20:04:38.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:38.167+0000 7f2832ffd700 1 --2- 172.21.15.146:0/3917418412 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28340f5410 0x7f2834109430 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f2828013510 tx=0x7f282800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:38.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:38.167+0000 7f281b7fe700 1 -- 172.21.15.146:0/3917418412 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2828009320 con 0x7f28340f5410 2024-01-21T20:04:38.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:38.167+0000 7f2839634700 1 -- 172.21.15.146:0/3917418412 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2834060440 con 0x7f28340f5410 2024-01-21T20:04:38.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:38.167+0000 7f281b7fe700 1 -- 172.21.15.146:0/3917418412 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2828013f00 con 0x7f28340f5410 2024-01-21T20:04:38.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:38.167+0000 7f281b7fe700 1 -- 172.21.15.146:0/3917418412 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2828014d90 con 0x7f28340f5410 2024-01-21T20:04:38.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:38.167+0000 7f2839634700 1 -- 172.21.15.146:0/3917418412 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2834109ae0 con 0x7f28340f5410 2024-01-21T20:04:38.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:38.168+0000 7f281b7fe700 1 -- 172.21.15.146:0/3917418412 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2828014420 con 0x7f28340f5410 2024-01-21T20:04:38.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:38.168+0000 7f2839634700 1 -- 172.21.15.146:0/3917418412 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2834045ad0 con 0x7f28340f5410 2024-01-21T20:04:38.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:38.168+0000 7f281b7fe700 1 --2- 172.21.15.146:0/3917418412 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f281c02ff60 0x7f281c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:38.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:38.168+0000 7f281b7fe700 1 -- 172.21.15.146:0/3917418412 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f28280410c0 con 0x7f28340f5410 2024-01-21T20:04:38.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:38.168+0000 7f28327fc700 1 --2- 172.21.15.146:0/3917418412 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f281c02ff60 0x7f281c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:38.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:38.169+0000 7f28327fc700 1 --2- 172.21.15.146:0/3917418412 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f281c02ff60 0x7f281c032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f28240096a0 tx=0x7f2824006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:38.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:38.173+0000 7f281b7fe700 1 -- 172.21.15.146:0/3917418412 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f28280095f0 con 0x7f28340f5410 2024-01-21T20:04:38.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:38.376+0000 7f2839634700 1 -- 172.21.15.146:0/3917418412 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f28340fae60 con 0x7f28340f5410 2024-01-21T20:04:38.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:38.376+0000 7f281b7fe700 1 -- 172.21.15.146:0/3917418412 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f282801db70 con 0x7f28340f5410 2024-01-21T20:04:38.367 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:04:38.367 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:04:38.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:38.378+0000 7f2839634700 1 -- 172.21.15.146:0/3917418412 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f281c02ff60 msgr2=0x7f281c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:38.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:38.378+0000 7f2839634700 1 --2- 172.21.15.146:0/3917418412 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f281c02ff60 0x7f281c032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f28240096a0 tx=0x7f2824006b40).stop 2024-01-21T20:04:38.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:38.378+0000 7f2839634700 1 -- 172.21.15.146:0/3917418412 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28340f5410 msgr2=0x7f2834109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:38.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:38.378+0000 7f2839634700 1 --2- 172.21.15.146:0/3917418412 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28340f5410 0x7f2834109430 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f2828013510 tx=0x7f282800b040).stop 2024-01-21T20:04:38.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:38.378+0000 7f2839634700 1 -- 172.21.15.146:0/3917418412 shutdown_connections 2024-01-21T20:04:38.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:38.378+0000 7f2839634700 1 --2- 172.21.15.146:0/3917418412 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f281c02ff60 0x7f281c032420 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:38.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:38.378+0000 7f2839634700 1 --2- 172.21.15.146:0/3917418412 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28340f5410 0x7f2834109430 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:38.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:38.378+0000 7f2839634700 1 -- 172.21.15.146:0/3917418412 >> 172.21.15.146:0/3917418412 conn(0x7f28340f0b70 msgr2=0x7f28340f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:38.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:38.378+0000 7f2839634700 1 -- 172.21.15.146:0/3917418412 shutdown_connections 2024-01-21T20:04:38.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:38.378+0000 7f2839634700 1 -- 172.21.15.146:0/3917418412 wait complete. 2024-01-21T20:04:38.369 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:04:38.828 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:38 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:04:37.177237+0000 mgr.smithi053.fgsmol (mgr.14164) 169 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:38.828 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:38 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:04:38.376851+0000 mon.smithi053 (mon.0) 240 : audit [DBG] from='client.? 172.21.15.146:0/3917418412' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:04:39.762 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:04:39.762 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:04:41.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:41.021+0000 7f1d2e9bd700 1 -- 172.21.15.146:0/1722720859 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d280f5410 msgr2=0x7f1d280f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:41.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:41.021+0000 7f1d2e9bd700 1 --2- 172.21.15.146:0/1722720859 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d280f5410 0x7f1d280f5830 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f1d18009fb0 tx=0x7f1d18009b30).stop 2024-01-21T20:04:41.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:41.021+0000 7f1d2e9bd700 1 -- 172.21.15.146:0/1722720859 shutdown_connections 2024-01-21T20:04:41.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:41.021+0000 7f1d2e9bd700 1 --2- 172.21.15.146:0/1722720859 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d280f5410 0x7f1d280f5830 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:41.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:41.021+0000 7f1d2e9bd700 1 -- 172.21.15.146:0/1722720859 >> 172.21.15.146:0/1722720859 conn(0x7f1d280f0b70 msgr2=0x7f1d280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:41.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:41.021+0000 7f1d2e9bd700 1 -- 172.21.15.146:0/1722720859 shutdown_connections 2024-01-21T20:04:41.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:41.021+0000 7f1d2e9bd700 1 -- 172.21.15.146:0/1722720859 wait complete. 2024-01-21T20:04:41.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:41.022+0000 7f1d2e9bd700 1 Processor -- start 2024-01-21T20:04:41.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:41.022+0000 7f1d2e9bd700 1 -- start start 2024-01-21T20:04:41.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:41.022+0000 7f1d2e9bd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d280f5410 0x7f1d28105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:41.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:41.022+0000 7f1d2e9bd700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d28105b20 con 0x7f1d280f5410 2024-01-21T20:04:41.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:41.022+0000 7f1d27fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d280f5410 0x7f1d28105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:41.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:41.022+0000 7f1d27fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d280f5410 0x7f1d28105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:56834/0 (socket says 172.21.15.146:56834) 2024-01-21T20:04:41.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:41.022+0000 7f1d27fff700 1 -- 172.21.15.146:0/918182730 learned_addr learned my addr 172.21.15.146:0/918182730 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:04:41.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:41.023+0000 7f1d27fff700 1 -- 172.21.15.146:0/918182730 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d18005040 con 0x7f1d280f5410 2024-01-21T20:04:41.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:41.023+0000 7f1d27fff700 1 --2- 172.21.15.146:0/918182730 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d280f5410 0x7f1d28105630 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f1d18013510 tx=0x7f1d1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:41.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:41.023+0000 7f1d24ff9700 1 -- 172.21.15.146:0/918182730 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d18009320 con 0x7f1d280f5410 2024-01-21T20:04:41.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:41.023+0000 7f1d2e9bd700 1 -- 172.21.15.146:0/918182730 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d28105d20 con 0x7f1d280f5410 2024-01-21T20:04:41.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:41.023+0000 7f1d24ff9700 1 -- 172.21.15.146:0/918182730 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1d18013f00 con 0x7f1d280f5410 2024-01-21T20:04:41.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:41.023+0000 7f1d24ff9700 1 -- 172.21.15.146:0/918182730 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d18014d90 con 0x7f1d280f5410 2024-01-21T20:04:41.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:41.023+0000 7f1d2e9bd700 1 -- 172.21.15.146:0/918182730 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d281061b0 con 0x7f1d280f5410 2024-01-21T20:04:41.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:41.024+0000 7f1d2e9bd700 1 -- 172.21.15.146:0/918182730 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d28045ad0 con 0x7f1d280f5410 2024-01-21T20:04:41.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:41.024+0000 7f1d24ff9700 1 -- 172.21.15.146:0/918182730 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1d180095f0 con 0x7f1d280f5410 2024-01-21T20:04:41.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:41.024+0000 7f1d24ff9700 1 --2- 172.21.15.146:0/918182730 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f1d1002ffb0 0x7f1d10032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:41.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:41.024+0000 7f1d24ff9700 1 -- 172.21.15.146:0/918182730 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1d18042d50 con 0x7f1d280f5410 2024-01-21T20:04:41.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:41.024+0000 7f1d277fe700 1 --2- 172.21.15.146:0/918182730 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f1d1002ffb0 0x7f1d10032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:41.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:41.025+0000 7f1d277fe700 1 --2- 172.21.15.146:0/918182730 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f1d1002ffb0 0x7f1d10032470 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f1d1c0096a0 tx=0x7f1d1c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:41.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:41.029+0000 7f1d24ff9700 1 -- 172.21.15.146:0/918182730 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1d18014420 con 0x7f1d280f5410 2024-01-21T20:04:41.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:40 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:04:39.177450+0000 mgr.smithi053.fgsmol (mgr.14164) 170 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:41.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:40 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:04:39.795313+0000 mon.smithi053 (mon.0) 241 : audit [INF] from='mgr.14164 172.21.15.53:0/2331671680' entity='mgr.smithi053.fgsmol' 2024-01-21T20:04:41.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:40 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cephadm 2024-01-21T20:04:39.797280+0000 mgr.smithi053.fgsmol (mgr.14164) 171 : cephadm [INF] Deploying daemon node-exporter.smithi053 on smithi053 2024-01-21T20:04:41.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:41.231+0000 7f1d2e9bd700 1 -- 172.21.15.146:0/918182730 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1d280fae60 con 0x7f1d280f5410 2024-01-21T20:04:41.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:41.232+0000 7f1d24ff9700 1 -- 172.21.15.146:0/918182730 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1d18011290 con 0x7f1d280f5410 2024-01-21T20:04:41.222 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:04:41.222 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:04:41.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:41.234+0000 7f1d2e9bd700 1 -- 172.21.15.146:0/918182730 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f1d1002ffb0 msgr2=0x7f1d10032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:41.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:41.234+0000 7f1d2e9bd700 1 --2- 172.21.15.146:0/918182730 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f1d1002ffb0 0x7f1d10032470 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f1d1c0096a0 tx=0x7f1d1c006b40).stop 2024-01-21T20:04:41.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:41.234+0000 7f1d2e9bd700 1 -- 172.21.15.146:0/918182730 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d280f5410 msgr2=0x7f1d28105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:41.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:41.234+0000 7f1d2e9bd700 1 --2- 172.21.15.146:0/918182730 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d280f5410 0x7f1d28105630 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f1d18013510 tx=0x7f1d1800b040).stop 2024-01-21T20:04:41.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:41.234+0000 7f1d2e9bd700 1 -- 172.21.15.146:0/918182730 shutdown_connections 2024-01-21T20:04:41.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:41.234+0000 7f1d2e9bd700 1 --2- 172.21.15.146:0/918182730 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f1d1002ffb0 0x7f1d10032470 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:41.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:41.234+0000 7f1d2e9bd700 1 --2- 172.21.15.146:0/918182730 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d280f5410 0x7f1d28105630 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:41.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:41.234+0000 7f1d2e9bd700 1 -- 172.21.15.146:0/918182730 >> 172.21.15.146:0/918182730 conn(0x7f1d280f0b70 msgr2=0x7f1d280f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:41.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:41.234+0000 7f1d2e9bd700 1 -- 172.21.15.146:0/918182730 shutdown_connections 2024-01-21T20:04:41.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:41.234+0000 7f1d2e9bd700 1 -- 172.21.15.146:0/918182730 wait complete. 2024-01-21T20:04:41.224 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:04:42.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:04:41.232511+0000 mon.smithi053 (mon.0) 242 : audit [DBG] from='client.? 172.21.15.146:0/918182730' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:04:42.604 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:04:42.604 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:04:43.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:42 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:04:41.177728+0000 mgr.smithi053.fgsmol (mgr.14164) 172 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:43.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:43.845+0000 7fe8e1274700 1 -- 172.21.15.146:0/2550500886 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8dc0f59d0 msgr2=0x7fe8dc0f5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:43.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:43.845+0000 7fe8e1274700 1 --2- 172.21.15.146:0/2550500886 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8dc0f59d0 0x7fe8dc0f5df0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7fe8d0009fb0 tx=0x7fe8d0009b30).stop 2024-01-21T20:04:43.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:43.845+0000 7fe8e1274700 1 -- 172.21.15.146:0/2550500886 shutdown_connections 2024-01-21T20:04:43.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:43.845+0000 7fe8e1274700 1 --2- 172.21.15.146:0/2550500886 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8dc0f59d0 0x7fe8dc0f5df0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:43.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:43.845+0000 7fe8e1274700 1 -- 172.21.15.146:0/2550500886 >> 172.21.15.146:0/2550500886 conn(0x7fe8dc0f02e0 msgr2=0x7fe8dc0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:43.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:43.845+0000 7fe8e1274700 1 -- 172.21.15.146:0/2550500886 shutdown_connections 2024-01-21T20:04:43.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:43.845+0000 7fe8e1274700 1 -- 172.21.15.146:0/2550500886 wait complete. 2024-01-21T20:04:43.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:43.846+0000 7fe8e1274700 1 Processor -- start 2024-01-21T20:04:43.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:43.846+0000 7fe8e1274700 1 -- start start 2024-01-21T20:04:43.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:43.846+0000 7fe8e1274700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8dc0f59d0 0x7fe8dc1093c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:43.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:43.846+0000 7fe8e1274700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8dc0f53b0 con 0x7fe8dc0f59d0 2024-01-21T20:04:43.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:43.846+0000 7fe8daffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8dc0f59d0 0x7fe8dc1093c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:43.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:43.846+0000 7fe8daffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8dc0f59d0 0x7fe8dc1093c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:56852/0 (socket says 172.21.15.146:56852) 2024-01-21T20:04:43.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:43.846+0000 7fe8daffd700 1 -- 172.21.15.146:0/2509752306 learned_addr learned my addr 172.21.15.146:0/2509752306 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:04:43.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:43.847+0000 7fe8daffd700 1 -- 172.21.15.146:0/2509752306 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d0005040 con 0x7fe8dc0f59d0 2024-01-21T20:04:43.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:43.847+0000 7fe8daffd700 1 --2- 172.21.15.146:0/2509752306 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8dc0f59d0 0x7fe8dc1093c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7fe8d0013510 tx=0x7fe8d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:43.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:43.847+0000 7fe8c37fe700 1 -- 172.21.15.146:0/2509752306 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8d0009320 con 0x7fe8dc0f59d0 2024-01-21T20:04:43.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:43.847+0000 7fe8e1274700 1 -- 172.21.15.146:0/2509752306 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8dc0f55b0 con 0x7fe8dc0f59d0 2024-01-21T20:04:43.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:43.847+0000 7fe8c37fe700 1 -- 172.21.15.146:0/2509752306 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe8d0013f00 con 0x7fe8dc0f59d0 2024-01-21T20:04:43.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:43.847+0000 7fe8c37fe700 1 -- 172.21.15.146:0/2509752306 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8d0014d90 con 0x7fe8dc0f59d0 2024-01-21T20:04:43.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:43.847+0000 7fe8e1274700 1 -- 172.21.15.146:0/2509752306 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8dc109a70 con 0x7fe8dc0f59d0 2024-01-21T20:04:43.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:43.848+0000 7fe8c37fe700 1 -- 172.21.15.146:0/2509752306 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe8d00095f0 con 0x7fe8dc0f59d0 2024-01-21T20:04:43.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:43.848+0000 7fe8e1274700 1 -- 172.21.15.146:0/2509752306 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8dc045ad0 con 0x7fe8dc0f59d0 2024-01-21T20:04:43.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:43.848+0000 7fe8c37fe700 1 --2- 172.21.15.146:0/2509752306 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fe8c402ff60 0x7fe8c4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:43.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:43.848+0000 7fe8c37fe700 1 -- 172.21.15.146:0/2509752306 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe8d00410a0 con 0x7fe8dc0f59d0 2024-01-21T20:04:43.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:43.848+0000 7fe8da7fc700 1 --2- 172.21.15.146:0/2509752306 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fe8c402ff60 0x7fe8c4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:43.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:43.849+0000 7fe8da7fc700 1 --2- 172.21.15.146:0/2509752306 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fe8c402ff60 0x7fe8c4032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fe8cc0096a0 tx=0x7fe8cc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:43.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:43.853+0000 7fe8c37fe700 1 -- 172.21.15.146:0/2509752306 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe8d0014420 con 0x7fe8dc0f59d0 2024-01-21T20:04:44.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:44.057+0000 7fe8e1274700 1 -- 172.21.15.146:0/2509752306 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe8dc10a290 con 0x7fe8dc0f59d0 2024-01-21T20:04:44.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:44.058+0000 7fe8c37fe700 1 -- 172.21.15.146:0/2509752306 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe8d0010230 con 0x7fe8dc0f59d0 2024-01-21T20:04:44.048 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:04:44.048 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:04:44.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:44.059+0000 7fe8e1274700 1 -- 172.21.15.146:0/2509752306 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fe8c402ff60 msgr2=0x7fe8c4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:44.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:44.059+0000 7fe8e1274700 1 --2- 172.21.15.146:0/2509752306 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fe8c402ff60 0x7fe8c4032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fe8cc0096a0 tx=0x7fe8cc006b40).stop 2024-01-21T20:04:44.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:44.059+0000 7fe8e1274700 1 -- 172.21.15.146:0/2509752306 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8dc0f59d0 msgr2=0x7fe8dc1093c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:44.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:44.059+0000 7fe8e1274700 1 --2- 172.21.15.146:0/2509752306 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8dc0f59d0 0x7fe8dc1093c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7fe8d0013510 tx=0x7fe8d000b040).stop 2024-01-21T20:04:44.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:44.059+0000 7fe8e1274700 1 -- 172.21.15.146:0/2509752306 shutdown_connections 2024-01-21T20:04:44.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:44.059+0000 7fe8e1274700 1 --2- 172.21.15.146:0/2509752306 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fe8c402ff60 0x7fe8c4032420 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:44.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:44.059+0000 7fe8e1274700 1 --2- 172.21.15.146:0/2509752306 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8dc0f59d0 0x7fe8dc1093c0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:44.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:44.059+0000 7fe8e1274700 1 -- 172.21.15.146:0/2509752306 >> 172.21.15.146:0/2509752306 conn(0x7fe8dc0f02e0 msgr2=0x7fe8dc0f0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:44.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:44.059+0000 7fe8e1274700 1 -- 172.21.15.146:0/2509752306 shutdown_connections 2024-01-21T20:04:44.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:44.059+0000 7fe8e1274700 1 -- 172.21.15.146:0/2509752306 wait complete. 2024-01-21T20:04:44.050 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:04:45.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:44 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:04:43.178022+0000 mgr.smithi053.fgsmol (mgr.14164) 173 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:45.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:44 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:04:44.057878+0000 mon.smithi053 (mon.0) 243 : audit [DBG] from='client.? 172.21.15.146:0/2509752306' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:04:45.452 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:04:45.453 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:04:46.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:46.781+0000 7f48d18b4700 1 -- 172.21.15.146:0/3613743599 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48cc0f4740 msgr2=0x7f48cc0f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:46.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:46.781+0000 7f48d18b4700 1 --2- 172.21.15.146:0/3613743599 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48cc0f4740 0x7f48cc0f4b60 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f48c0009fb0 tx=0x7f48c0009b30).stop 2024-01-21T20:04:46.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:46.781+0000 7f48d18b4700 1 -- 172.21.15.146:0/3613743599 shutdown_connections 2024-01-21T20:04:46.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:46.781+0000 7f48d18b4700 1 --2- 172.21.15.146:0/3613743599 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48cc0f4740 0x7f48cc0f4b60 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:46.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:46.781+0000 7f48d18b4700 1 -- 172.21.15.146:0/3613743599 >> 172.21.15.146:0/3613743599 conn(0x7f48cc0f0260 msgr2=0x7f48cc0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:46.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:46.781+0000 7f48d18b4700 1 -- 172.21.15.146:0/3613743599 shutdown_connections 2024-01-21T20:04:46.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:46.781+0000 7f48d18b4700 1 -- 172.21.15.146:0/3613743599 wait complete. 2024-01-21T20:04:46.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:46.782+0000 7f48d18b4700 1 Processor -- start 2024-01-21T20:04:46.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:46.782+0000 7f48d18b4700 1 -- start start 2024-01-21T20:04:46.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:46.782+0000 7f48d18b4700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48cc1067f0 0x7f48cc106c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:46.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:46.782+0000 7f48d18b4700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48cc107100 con 0x7f48cc1067f0 2024-01-21T20:04:46.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:46.782+0000 7f48caffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48cc1067f0 0x7f48cc106c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:46.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:46.782+0000 7f48caffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48cc1067f0 0x7f48cc106c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:56866/0 (socket says 172.21.15.146:56866) 2024-01-21T20:04:46.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:46.782+0000 7f48caffd700 1 -- 172.21.15.146:0/2402076686 learned_addr learned my addr 172.21.15.146:0/2402076686 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:04:46.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:46.782+0000 7f48caffd700 1 -- 172.21.15.146:0/2402076686 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c0005040 con 0x7f48cc1067f0 2024-01-21T20:04:46.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:46.782+0000 7f48caffd700 1 --2- 172.21.15.146:0/2402076686 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48cc1067f0 0x7f48cc106c10 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f48c0009f80 tx=0x7f48c0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:46.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:46.783+0000 7f48b37fe700 1 -- 172.21.15.146:0/2402076686 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48c00136c0 con 0x7f48cc1067f0 2024-01-21T20:04:46.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:46.783+0000 7f48d18b4700 1 -- 172.21.15.146:0/2402076686 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48cc107300 con 0x7f48cc1067f0 2024-01-21T20:04:46.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:46.783+0000 7f48b37fe700 1 -- 172.21.15.146:0/2402076686 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f48c00095f0 con 0x7f48cc1067f0 2024-01-21T20:04:46.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:46.783+0000 7f48b37fe700 1 -- 172.21.15.146:0/2402076686 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48c0014420 con 0x7f48cc1067f0 2024-01-21T20:04:46.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:46.783+0000 7f48d18b4700 1 -- 172.21.15.146:0/2402076686 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48cc107e50 con 0x7f48cc1067f0 2024-01-21T20:04:46.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:46.783+0000 7f48b37fe700 1 -- 172.21.15.146:0/2402076686 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f48c0025060 con 0x7f48cc1067f0 2024-01-21T20:04:46.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:46.783+0000 7f48d18b4700 1 -- 172.21.15.146:0/2402076686 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48cc059100 con 0x7f48cc1067f0 2024-01-21T20:04:46.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:46.784+0000 7f48b37fe700 1 --2- 172.21.15.146:0/2402076686 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f48b402ff60 0x7f48b4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:46.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:46.784+0000 7f48b37fe700 1 -- 172.21.15.146:0/2402076686 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f48c0040e60 con 0x7f48cc1067f0 2024-01-21T20:04:46.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:46.784+0000 7f48ca7fc700 1 --2- 172.21.15.146:0/2402076686 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f48b402ff60 0x7f48b4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:46.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:46.784+0000 7f48ca7fc700 1 --2- 172.21.15.146:0/2402076686 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f48b402ff60 0x7f48b4032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f48bc0096a0 tx=0x7f48bc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:46.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:46.788+0000 7f48b37fe700 1 -- 172.21.15.146:0/2402076686 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f48c001cb60 con 0x7f48cc1067f0 2024-01-21T20:04:46.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:46.996+0000 7f48d18b4700 1 -- 172.21.15.146:0/2402076686 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f48cc045ad0 con 0x7f48cc1067f0 2024-01-21T20:04:46.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:46.996+0000 7f48b37fe700 1 -- 172.21.15.146:0/2402076686 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f48c000f280 con 0x7f48cc1067f0 2024-01-21T20:04:46.986 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:04:46.987 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:04:46.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:46.998+0000 7f48d18b4700 1 -- 172.21.15.146:0/2402076686 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f48b402ff60 msgr2=0x7f48b4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:46.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:46.998+0000 7f48d18b4700 1 --2- 172.21.15.146:0/2402076686 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f48b402ff60 0x7f48b4032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f48bc0096a0 tx=0x7f48bc006b40).stop 2024-01-21T20:04:46.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:46.998+0000 7f48d18b4700 1 -- 172.21.15.146:0/2402076686 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48cc1067f0 msgr2=0x7f48cc106c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:46.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:46.998+0000 7f48d18b4700 1 --2- 172.21.15.146:0/2402076686 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48cc1067f0 0x7f48cc106c10 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f48c0009f80 tx=0x7f48c0009b30).stop 2024-01-21T20:04:46.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:46.998+0000 7f48d18b4700 1 -- 172.21.15.146:0/2402076686 shutdown_connections 2024-01-21T20:04:46.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:46.998+0000 7f48d18b4700 1 --2- 172.21.15.146:0/2402076686 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f48b402ff60 0x7f48b4032420 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:46.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:46.998+0000 7f48d18b4700 1 --2- 172.21.15.146:0/2402076686 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48cc1067f0 0x7f48cc106c10 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:46.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:46.998+0000 7f48d18b4700 1 -- 172.21.15.146:0/2402076686 >> 172.21.15.146:0/2402076686 conn(0x7f48cc0f0260 msgr2=0x7f48cc0f0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:46.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:46.998+0000 7f48d18b4700 1 -- 172.21.15.146:0/2402076686 shutdown_connections 2024-01-21T20:04:46.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:46.998+0000 7f48d18b4700 1 -- 172.21.15.146:0/2402076686 wait complete. 2024-01-21T20:04:46.989 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:04:47.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:46 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:04:45.178313+0000 mgr.smithi053.fgsmol (mgr.14164) 174 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:48.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:47 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:04:46.996795+0000 mon.smithi053 (mon.0) 244 : audit [DBG] from='client.? 172.21.15.146:0/2402076686' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:04:48.412 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:04:48.412 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:04:49.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:48 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:04:47.178664+0000 mgr.smithi053.fgsmol (mgr.14164) 175 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:49.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:49.678+0000 7f6fe7cb3700 1 -- 172.21.15.146:0/3839591777 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6fe00f6510 msgr2=0x7f6fe00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:49.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:49.678+0000 7f6fe7cb3700 1 --2- 172.21.15.146:0/3839591777 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6fe00f6510 0x7f6fe00f6930 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f6fd0009fb0 tx=0x7f6fd0009b30).stop 2024-01-21T20:04:49.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:49.678+0000 7f6fe7cb3700 1 -- 172.21.15.146:0/3839591777 shutdown_connections 2024-01-21T20:04:49.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:49.678+0000 7f6fe7cb3700 1 --2- 172.21.15.146:0/3839591777 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6fe00f6510 0x7f6fe00f6930 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:49.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:49.678+0000 7f6fe7cb3700 1 -- 172.21.15.146:0/3839591777 >> 172.21.15.146:0/3839591777 conn(0x7f6fe00f1c70 msgr2=0x7f6fe00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:49.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:49.679+0000 7f6fe7cb3700 1 -- 172.21.15.146:0/3839591777 shutdown_connections 2024-01-21T20:04:49.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:49.679+0000 7f6fe7cb3700 1 -- 172.21.15.146:0/3839591777 wait complete. 2024-01-21T20:04:49.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:49.679+0000 7f6fe7cb3700 1 Processor -- start 2024-01-21T20:04:49.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:49.679+0000 7f6fe7cb3700 1 -- start start 2024-01-21T20:04:49.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:49.679+0000 7f6fe7cb3700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6fe00f6510 0x7f6fe0108910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:49.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:49.679+0000 7f6fe7cb3700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe0108e00 con 0x7f6fe00f6510 2024-01-21T20:04:49.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:49.679+0000 7f6fe5a4f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6fe00f6510 0x7f6fe0108910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:49.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:49.679+0000 7f6fe5a4f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6fe00f6510 0x7f6fe0108910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:53506/0 (socket says 172.21.15.146:53506) 2024-01-21T20:04:49.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:49.679+0000 7f6fe5a4f700 1 -- 172.21.15.146:0/3201187739 learned_addr learned my addr 172.21.15.146:0/3201187739 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:04:49.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:49.680+0000 7f6fe5a4f700 1 -- 172.21.15.146:0/3201187739 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd0005040 con 0x7f6fe00f6510 2024-01-21T20:04:49.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:49.680+0000 7f6fe5a4f700 1 --2- 172.21.15.146:0/3201187739 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6fe00f6510 0x7f6fe0108910 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f6fd0013510 tx=0x7f6fd000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:49.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:49.680+0000 7f6fd67fc700 1 -- 172.21.15.146:0/3201187739 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fd0009320 con 0x7f6fe00f6510 2024-01-21T20:04:49.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:49.680+0000 7f6fe7cb3700 1 -- 172.21.15.146:0/3201187739 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fe0109000 con 0x7f6fe00f6510 2024-01-21T20:04:49.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:49.680+0000 7f6fd67fc700 1 -- 172.21.15.146:0/3201187739 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6fd0013f00 con 0x7f6fe00f6510 2024-01-21T20:04:49.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:49.680+0000 7f6fd67fc700 1 -- 172.21.15.146:0/3201187739 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fd0014d90 con 0x7f6fe00f6510 2024-01-21T20:04:49.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:49.680+0000 7f6fe7cb3700 1 -- 172.21.15.146:0/3201187739 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fe0109490 con 0x7f6fe00f6510 2024-01-21T20:04:49.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:49.681+0000 7f6fd67fc700 1 -- 172.21.15.146:0/3201187739 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6fd00095f0 con 0x7f6fe00f6510 2024-01-21T20:04:49.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:49.681+0000 7f6fe7cb3700 1 -- 172.21.15.146:0/3201187739 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fe0045ad0 con 0x7f6fe00f6510 2024-01-21T20:04:49.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:49.681+0000 7f6fd67fc700 1 --2- 172.21.15.146:0/3201187739 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f6fcc02ff60 0x7f6fcc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:49.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:49.681+0000 7f6fd67fc700 1 -- 172.21.15.146:0/3201187739 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6fd00410a0 con 0x7f6fe00f6510 2024-01-21T20:04:49.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:49.681+0000 7f6fe524e700 1 --2- 172.21.15.146:0/3201187739 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f6fcc02ff60 0x7f6fcc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:49.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:49.682+0000 7f6fe524e700 1 --2- 172.21.15.146:0/3201187739 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f6fcc02ff60 0x7f6fcc032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f6fdc0096a0 tx=0x7f6fdc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:49.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:49.686+0000 7f6fd67fc700 1 -- 172.21.15.146:0/3201187739 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6fd001d7f0 con 0x7f6fe00f6510 2024-01-21T20:04:49.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:49.889+0000 7f6fe7cb3700 1 -- 172.21.15.146:0/3201187739 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6fe010a340 con 0x7f6fe00f6510 2024-01-21T20:04:49.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:49.889+0000 7f6fd67fc700 1 -- 172.21.15.146:0/3201187739 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6fd0010230 con 0x7f6fe00f6510 2024-01-21T20:04:49.880 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:04:49.880 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:04:49.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:49.891+0000 7f6fe7cb3700 1 -- 172.21.15.146:0/3201187739 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f6fcc02ff60 msgr2=0x7f6fcc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:49.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:49.891+0000 7f6fe7cb3700 1 --2- 172.21.15.146:0/3201187739 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f6fcc02ff60 0x7f6fcc032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f6fdc0096a0 tx=0x7f6fdc006b40).stop 2024-01-21T20:04:49.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:49.891+0000 7f6fe7cb3700 1 -- 172.21.15.146:0/3201187739 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6fe00f6510 msgr2=0x7f6fe0108910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:49.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:49.891+0000 7f6fe7cb3700 1 --2- 172.21.15.146:0/3201187739 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6fe00f6510 0x7f6fe0108910 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f6fd0013510 tx=0x7f6fd000b040).stop 2024-01-21T20:04:49.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:49.891+0000 7f6fe7cb3700 1 -- 172.21.15.146:0/3201187739 shutdown_connections 2024-01-21T20:04:49.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:49.891+0000 7f6fe7cb3700 1 --2- 172.21.15.146:0/3201187739 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f6fcc02ff60 0x7f6fcc032420 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:49.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:49.891+0000 7f6fe7cb3700 1 --2- 172.21.15.146:0/3201187739 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6fe00f6510 0x7f6fe0108910 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:49.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:49.891+0000 7f6fe7cb3700 1 -- 172.21.15.146:0/3201187739 >> 172.21.15.146:0/3201187739 conn(0x7f6fe00f1c70 msgr2=0x7f6fe00f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:49.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:49.891+0000 7f6fe7cb3700 1 -- 172.21.15.146:0/3201187739 shutdown_connections 2024-01-21T20:04:49.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:49.891+0000 7f6fe7cb3700 1 -- 172.21.15.146:0/3201187739 wait complete. 2024-01-21T20:04:49.883 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:04:51.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:50 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:04:49.179014+0000 mgr.smithi053.fgsmol (mgr.14164) 176 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:51.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:50 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:04:49.890624+0000 mon.smithi053 (mon.0) 245 : audit [DBG] from='client.? 172.21.15.146:0/3201187739' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:04:51.307 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:04:51.307 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:04:52.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:52.511+0000 7f56bd310700 1 -- 172.21.15.146:0/3237187559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f56b80f5410 msgr2=0x7f56b80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:52.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:52.511+0000 7f56bd310700 1 --2- 172.21.15.146:0/3237187559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f56b80f5410 0x7f56b80f5830 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f56a8009fb0 tx=0x7f56a8009b30).stop 2024-01-21T20:04:52.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:52.512+0000 7f56bd310700 1 -- 172.21.15.146:0/3237187559 shutdown_connections 2024-01-21T20:04:52.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:52.512+0000 7f56bd310700 1 --2- 172.21.15.146:0/3237187559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f56b80f5410 0x7f56b80f5830 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:52.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:52.512+0000 7f56bd310700 1 -- 172.21.15.146:0/3237187559 >> 172.21.15.146:0/3237187559 conn(0x7f56b80f0b70 msgr2=0x7f56b80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:52.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:52.512+0000 7f56bd310700 1 -- 172.21.15.146:0/3237187559 shutdown_connections 2024-01-21T20:04:52.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:52.512+0000 7f56bd310700 1 -- 172.21.15.146:0/3237187559 wait complete. 2024-01-21T20:04:52.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:52.512+0000 7f56bd310700 1 Processor -- start 2024-01-21T20:04:52.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:52.512+0000 7f56bd310700 1 -- start start 2024-01-21T20:04:52.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:52.512+0000 7f56bd310700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f56b80f5410 0x7f56b8105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:52.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:52.512+0000 7f56bd310700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b8105af0 con 0x7f56b80f5410 2024-01-21T20:04:52.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:52.513+0000 7f56b6ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f56b80f5410 0x7f56b8105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:52.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:52.513+0000 7f56b6ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f56b80f5410 0x7f56b8105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:53522/0 (socket says 172.21.15.146:53522) 2024-01-21T20:04:52.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:52.513+0000 7f56b6ffd700 1 -- 172.21.15.146:0/2030389148 learned_addr learned my addr 172.21.15.146:0/2030389148 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:04:52.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:52.513+0000 7f56b6ffd700 1 -- 172.21.15.146:0/2030389148 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a8005040 con 0x7f56b80f5410 2024-01-21T20:04:52.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:52.513+0000 7f56b6ffd700 1 --2- 172.21.15.146:0/2030389148 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f56b80f5410 0x7f56b8105600 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f56a8013510 tx=0x7f56a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:52.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:52.513+0000 7f56af7fe700 1 -- 172.21.15.146:0/2030389148 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56a8009320 con 0x7f56b80f5410 2024-01-21T20:04:52.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:52.513+0000 7f56bd310700 1 -- 172.21.15.146:0/2030389148 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56b8105cf0 con 0x7f56b80f5410 2024-01-21T20:04:52.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:52.513+0000 7f56af7fe700 1 -- 172.21.15.146:0/2030389148 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f56a8013f00 con 0x7f56b80f5410 2024-01-21T20:04:52.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:52.513+0000 7f56af7fe700 1 -- 172.21.15.146:0/2030389148 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56a8014d90 con 0x7f56b80f5410 2024-01-21T20:04:52.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:52.513+0000 7f56bd310700 1 -- 172.21.15.146:0/2030389148 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56b8106180 con 0x7f56b80f5410 2024-01-21T20:04:52.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:52.514+0000 7f56af7fe700 1 -- 172.21.15.146:0/2030389148 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f56a80263a0 con 0x7f56b80f5410 2024-01-21T20:04:52.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:52.514+0000 7f56af7fe700 1 --2- 172.21.15.146:0/2030389148 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f56a402ff60 0x7f56a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:52.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:52.514+0000 7f56af7fe700 1 -- 172.21.15.146:0/2030389148 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f56a80410c0 con 0x7f56b80f5410 2024-01-21T20:04:52.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:52.514+0000 7f56bd310700 1 -- 172.21.15.146:0/2030389148 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56b8045ad0 con 0x7f56b80f5410 2024-01-21T20:04:52.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:52.515+0000 7f56b67fc700 1 --2- 172.21.15.146:0/2030389148 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f56a402ff60 0x7f56a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:52.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:52.515+0000 7f56b67fc700 1 --2- 172.21.15.146:0/2030389148 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f56a402ff60 0x7f56a4032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f56a00096a0 tx=0x7f56a0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:52.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:52.519+0000 7f56af7fe700 1 -- 172.21.15.146:0/2030389148 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f56a8012920 con 0x7f56b80f5410 2024-01-21T20:04:52.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:52.722+0000 7f56bd310700 1 -- 172.21.15.146:0/2030389148 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f56b80fae60 con 0x7f56b80f5410 2024-01-21T20:04:52.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:52.723+0000 7f56af7fe700 1 -- 172.21.15.146:0/2030389148 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f56a802b020 con 0x7f56b80f5410 2024-01-21T20:04:52.714 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:04:52.714 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:04:52.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:52.724+0000 7f56bd310700 1 -- 172.21.15.146:0/2030389148 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f56a402ff60 msgr2=0x7f56a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:52.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:52.724+0000 7f56bd310700 1 --2- 172.21.15.146:0/2030389148 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f56a402ff60 0x7f56a4032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f56a00096a0 tx=0x7f56a0006b40).stop 2024-01-21T20:04:52.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:52.724+0000 7f56bd310700 1 -- 172.21.15.146:0/2030389148 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f56b80f5410 msgr2=0x7f56b8105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:52.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:52.724+0000 7f56bd310700 1 --2- 172.21.15.146:0/2030389148 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f56b80f5410 0x7f56b8105600 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f56a8013510 tx=0x7f56a800b040).stop 2024-01-21T20:04:52.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:52.725+0000 7f56bd310700 1 -- 172.21.15.146:0/2030389148 shutdown_connections 2024-01-21T20:04:52.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:52.725+0000 7f56bd310700 1 --2- 172.21.15.146:0/2030389148 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f56a402ff60 0x7f56a4032420 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:52.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:52.725+0000 7f56bd310700 1 --2- 172.21.15.146:0/2030389148 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f56b80f5410 0x7f56b8105600 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:52.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:52.725+0000 7f56bd310700 1 -- 172.21.15.146:0/2030389148 >> 172.21.15.146:0/2030389148 conn(0x7f56b80f0b70 msgr2=0x7f56b80f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:52.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:52.725+0000 7f56bd310700 1 -- 172.21.15.146:0/2030389148 shutdown_connections 2024-01-21T20:04:52.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:52.725+0000 7f56bd310700 1 -- 172.21.15.146:0/2030389148 wait complete. 2024-01-21T20:04:52.717 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:04:53.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:52 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:04:51.179208+0000 mgr.smithi053.fgsmol (mgr.14164) 177 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:53.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:52 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:04:52.724499+0000 mon.smithi053 (mon.0) 246 : audit [DBG] from='client.? 172.21.15.146:0/2030389148' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:04:54.123 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:04:54.124 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:04:55.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:54 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:04:53.179495+0000 mgr.smithi053.fgsmol (mgr.14164) 178 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:55.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:55.342+0000 7f55fc4da700 1 -- 172.21.15.146:0/398875334 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55f40f6510 msgr2=0x7f55f40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:55.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:55.342+0000 7f55fc4da700 1 --2- 172.21.15.146:0/398875334 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55f40f6510 0x7f55f40f6930 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f55f0009fb0 tx=0x7f55f0009b30).stop 2024-01-21T20:04:55.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:55.342+0000 7f55fc4da700 1 -- 172.21.15.146:0/398875334 shutdown_connections 2024-01-21T20:04:55.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:55.342+0000 7f55fc4da700 1 --2- 172.21.15.146:0/398875334 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55f40f6510 0x7f55f40f6930 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:55.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:55.342+0000 7f55fc4da700 1 -- 172.21.15.146:0/398875334 >> 172.21.15.146:0/398875334 conn(0x7f55f40f1c70 msgr2=0x7f55f40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:55.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:55.342+0000 7f55fc4da700 1 -- 172.21.15.146:0/398875334 shutdown_connections 2024-01-21T20:04:55.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:55.342+0000 7f55fc4da700 1 -- 172.21.15.146:0/398875334 wait complete. 2024-01-21T20:04:55.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:55.342+0000 7f55fc4da700 1 Processor -- start 2024-01-21T20:04:55.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:55.343+0000 7f55fc4da700 1 -- start start 2024-01-21T20:04:55.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:55.343+0000 7f55fc4da700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55f40f6510 0x7f55f4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:55.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:55.343+0000 7f55fc4da700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f4106c20 con 0x7f55f40f6510 2024-01-21T20:04:55.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:55.343+0000 7f55fa276700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55f40f6510 0x7f55f4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:55.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:55.343+0000 7f55fa276700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55f40f6510 0x7f55f4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:53538/0 (socket says 172.21.15.146:53538) 2024-01-21T20:04:55.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:55.343+0000 7f55fa276700 1 -- 172.21.15.146:0/1517558559 learned_addr learned my addr 172.21.15.146:0/1517558559 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:04:55.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:55.343+0000 7f55fa276700 1 -- 172.21.15.146:0/1517558559 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55f0005040 con 0x7f55f40f6510 2024-01-21T20:04:55.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:55.343+0000 7f55fa276700 1 --2- 172.21.15.146:0/1517558559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55f40f6510 0x7f55f4106730 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f55f0013510 tx=0x7f55f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:55.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:55.344+0000 7f55eaffd700 1 -- 172.21.15.146:0/1517558559 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55f0009320 con 0x7f55f40f6510 2024-01-21T20:04:55.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:55.344+0000 7f55fc4da700 1 -- 172.21.15.146:0/1517558559 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55f4106e20 con 0x7f55f40f6510 2024-01-21T20:04:55.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:55.344+0000 7f55eaffd700 1 -- 172.21.15.146:0/1517558559 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f55f0013f00 con 0x7f55f40f6510 2024-01-21T20:04:55.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:55.344+0000 7f55eaffd700 1 -- 172.21.15.146:0/1517558559 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55f0014d90 con 0x7f55f40f6510 2024-01-21T20:04:55.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:55.344+0000 7f55fc4da700 1 -- 172.21.15.146:0/1517558559 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55f41072b0 con 0x7f55f40f6510 2024-01-21T20:04:55.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:55.344+0000 7f55eaffd700 1 -- 172.21.15.146:0/1517558559 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f55f00095f0 con 0x7f55f40f6510 2024-01-21T20:04:55.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:55.344+0000 7f55fc4da700 1 -- 172.21.15.146:0/1517558559 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55f4045ad0 con 0x7f55f40f6510 2024-01-21T20:04:55.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:55.345+0000 7f55eaffd700 1 --2- 172.21.15.146:0/1517558559 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f55e002ff60 0x7f55e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:55.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:55.345+0000 7f55eaffd700 1 -- 172.21.15.146:0/1517558559 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f55f0041000 con 0x7f55f40f6510 2024-01-21T20:04:55.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:55.345+0000 7f55f9a75700 1 --2- 172.21.15.146:0/1517558559 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f55e002ff60 0x7f55e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:55.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:55.345+0000 7f55f9a75700 1 --2- 172.21.15.146:0/1517558559 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f55e002ff60 0x7f55e0032420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f55e40096a0 tx=0x7f55e4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:55.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:55.349+0000 7f55eaffd700 1 -- 172.21.15.146:0/1517558559 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f55f0014420 con 0x7f55f40f6510 2024-01-21T20:04:55.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:55.556+0000 7f55fc4da700 1 -- 172.21.15.146:0/1517558559 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f55f40fbf60 con 0x7f55f40f6510 2024-01-21T20:04:55.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:55.556+0000 7f55eaffd700 1 -- 172.21.15.146:0/1517558559 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f55f0010230 con 0x7f55f40f6510 2024-01-21T20:04:55.547 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:04:55.547 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:04:55.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:55.558+0000 7f55fc4da700 1 -- 172.21.15.146:0/1517558559 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f55e002ff60 msgr2=0x7f55e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:55.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:55.558+0000 7f55fc4da700 1 --2- 172.21.15.146:0/1517558559 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f55e002ff60 0x7f55e0032420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f55e40096a0 tx=0x7f55e4006b40).stop 2024-01-21T20:04:55.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:55.558+0000 7f55fc4da700 1 -- 172.21.15.146:0/1517558559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55f40f6510 msgr2=0x7f55f4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:55.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:55.558+0000 7f55fc4da700 1 --2- 172.21.15.146:0/1517558559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55f40f6510 0x7f55f4106730 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f55f0013510 tx=0x7f55f000b040).stop 2024-01-21T20:04:55.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:55.558+0000 7f55fc4da700 1 -- 172.21.15.146:0/1517558559 shutdown_connections 2024-01-21T20:04:55.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:55.558+0000 7f55fc4da700 1 --2- 172.21.15.146:0/1517558559 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f55e002ff60 0x7f55e0032420 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:55.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:55.558+0000 7f55fc4da700 1 --2- 172.21.15.146:0/1517558559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55f40f6510 0x7f55f4106730 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:55.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:55.558+0000 7f55fc4da700 1 -- 172.21.15.146:0/1517558559 >> 172.21.15.146:0/1517558559 conn(0x7f55f40f1c70 msgr2=0x7f55f40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:55.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:55.558+0000 7f55fc4da700 1 -- 172.21.15.146:0/1517558559 shutdown_connections 2024-01-21T20:04:55.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:55.558+0000 7f55fc4da700 1 -- 172.21.15.146:0/1517558559 wait complete. 2024-01-21T20:04:55.550 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:04:56.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:55 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:04:55.557698+0000 mon.smithi053 (mon.0) 247 : audit [DBG] from='client.? 172.21.15.146:0/1517558559' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:04:57.011 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:04:57.011 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:04:57.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:56 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:04:55.179863+0000 mgr.smithi053.fgsmol (mgr.14164) 179 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:58.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:58.203+0000 7fdcdc01e700 1 -- 172.21.15.146:0/1313011975 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdcd40f5410 msgr2=0x7fdcd40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:58.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:58.203+0000 7fdcdc01e700 1 --2- 172.21.15.146:0/1313011975 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdcd40f5410 0x7fdcd40f5830 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7fdcc8009fb0 tx=0x7fdcc8009b30).stop 2024-01-21T20:04:58.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:58.203+0000 7fdcdc01e700 1 -- 172.21.15.146:0/1313011975 shutdown_connections 2024-01-21T20:04:58.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:58.203+0000 7fdcdc01e700 1 --2- 172.21.15.146:0/1313011975 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdcd40f5410 0x7fdcd40f5830 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:58.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:58.203+0000 7fdcdc01e700 1 -- 172.21.15.146:0/1313011975 >> 172.21.15.146:0/1313011975 conn(0x7fdcd40f0b70 msgr2=0x7fdcd40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:58.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:58.203+0000 7fdcdc01e700 1 -- 172.21.15.146:0/1313011975 shutdown_connections 2024-01-21T20:04:58.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:58.203+0000 7fdcdc01e700 1 -- 172.21.15.146:0/1313011975 wait complete. 2024-01-21T20:04:58.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:58.204+0000 7fdcdc01e700 1 Processor -- start 2024-01-21T20:04:58.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:58.204+0000 7fdcdc01e700 1 -- start start 2024-01-21T20:04:58.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:58.204+0000 7fdcdc01e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdcd40f5410 0x7fdcd40fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:58.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:58.204+0000 7fdcdc01e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd40fb820 con 0x7fdcd40f5410 2024-01-21T20:04:58.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:58.204+0000 7fdcd9dba700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdcd40f5410 0x7fdcd40fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:58.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:58.204+0000 7fdcd9dba700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdcd40f5410 0x7fdcd40fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:40980/0 (socket says 172.21.15.146:40980) 2024-01-21T20:04:58.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:58.204+0000 7fdcd9dba700 1 -- 172.21.15.146:0/1369633508 learned_addr learned my addr 172.21.15.146:0/1369633508 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:04:58.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:58.205+0000 7fdcd9dba700 1 -- 172.21.15.146:0/1369633508 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc8005040 con 0x7fdcd40f5410 2024-01-21T20:04:58.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:58.205+0000 7fdcd9dba700 1 --2- 172.21.15.146:0/1369633508 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdcd40f5410 0x7fdcd40fd3a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7fdcc8013510 tx=0x7fdcc800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:58.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:58.205+0000 7fdcc67fc700 1 -- 172.21.15.146:0/1369633508 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdcc8009320 con 0x7fdcd40f5410 2024-01-21T20:04:58.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:58.205+0000 7fdcdc01e700 1 -- 172.21.15.146:0/1369633508 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcd40fb9c0 con 0x7fdcd40f5410 2024-01-21T20:04:58.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:58.205+0000 7fdcc67fc700 1 -- 172.21.15.146:0/1369633508 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdcc8013f00 con 0x7fdcd40f5410 2024-01-21T20:04:58.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:58.205+0000 7fdcc67fc700 1 -- 172.21.15.146:0/1369633508 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdcc8014d90 con 0x7fdcd40f5410 2024-01-21T20:04:58.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:58.205+0000 7fdcdc01e700 1 -- 172.21.15.146:0/1369633508 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcd40fbe50 con 0x7fdcd40f5410 2024-01-21T20:04:58.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:58.206+0000 7fdcdc01e700 1 -- 172.21.15.146:0/1369633508 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcd4059100 con 0x7fdcd40f5410 2024-01-21T20:04:58.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:58.206+0000 7fdcc67fc700 1 -- 172.21.15.146:0/1369633508 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdcc8014420 con 0x7fdcd40f5410 2024-01-21T20:04:58.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:58.206+0000 7fdcc67fc700 1 --2- 172.21.15.146:0/1369633508 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fdcc002ff60 0x7fdcc0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:58.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:58.206+0000 7fdcc67fc700 1 -- 172.21.15.146:0/1369633508 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdcc8040fb0 con 0x7fdcd40f5410 2024-01-21T20:04:58.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:58.206+0000 7fdcd95b9700 1 --2- 172.21.15.146:0/1369633508 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fdcc002ff60 0x7fdcc0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:58.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:58.207+0000 7fdcd95b9700 1 --2- 172.21.15.146:0/1369633508 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fdcc002ff60 0x7fdcc0032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fdcd00096a0 tx=0x7fdcd0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:58.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:58.211+0000 7fdcc67fc700 1 -- 172.21.15.146:0/1369633508 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdcc80095f0 con 0x7fdcd40f5410 2024-01-21T20:04:58.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:58.418+0000 7fdcdc01e700 1 -- 172.21.15.146:0/1369633508 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdcd40fae60 con 0x7fdcd40f5410 2024-01-21T20:04:58.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:58.418+0000 7fdcc67fc700 1 -- 172.21.15.146:0/1369633508 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdcc801db70 con 0x7fdcd40f5410 2024-01-21T20:04:58.409 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:04:58.409 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:04:58.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:58.420+0000 7fdcdc01e700 1 -- 172.21.15.146:0/1369633508 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fdcc002ff60 msgr2=0x7fdcc0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:58.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:58.420+0000 7fdcdc01e700 1 --2- 172.21.15.146:0/1369633508 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fdcc002ff60 0x7fdcc0032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fdcd00096a0 tx=0x7fdcd0006b40).stop 2024-01-21T20:04:58.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:58.420+0000 7fdcdc01e700 1 -- 172.21.15.146:0/1369633508 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdcd40f5410 msgr2=0x7fdcd40fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:58.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:58.420+0000 7fdcdc01e700 1 --2- 172.21.15.146:0/1369633508 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdcd40f5410 0x7fdcd40fd3a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7fdcc8013510 tx=0x7fdcc800b040).stop 2024-01-21T20:04:58.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:58.420+0000 7fdcdc01e700 1 -- 172.21.15.146:0/1369633508 shutdown_connections 2024-01-21T20:04:58.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:58.420+0000 7fdcdc01e700 1 --2- 172.21.15.146:0/1369633508 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fdcc002ff60 0x7fdcc0032420 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:58.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:58.420+0000 7fdcdc01e700 1 --2- 172.21.15.146:0/1369633508 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdcd40f5410 0x7fdcd40fd3a0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:58.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:58.420+0000 7fdcdc01e700 1 -- 172.21.15.146:0/1369633508 >> 172.21.15.146:0/1369633508 conn(0x7fdcd40f0b70 msgr2=0x7fdcd405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:58.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:58.420+0000 7fdcdc01e700 1 -- 172.21.15.146:0/1369633508 shutdown_connections 2024-01-21T20:04:58.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:04:58.420+0000 7fdcdc01e700 1 -- 172.21.15.146:0/1369633508 wait complete. 2024-01-21T20:04:58.412 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:04:59.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:58 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:04:57.180235+0000 mgr.smithi053.fgsmol (mgr.14164) 180 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:59.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:04:58 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:04:58.419730+0000 mon.smithi053 (mon.0) 248 : audit [DBG] from='client.? 172.21.15.146:0/1369633508' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:04:59.828 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:04:59.828 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:05:01.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:01.041+0000 7feecb4aa700 1 -- 172.21.15.146:0/3098352081 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feec406b960 msgr2=0x7feec4069e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:01.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:01.041+0000 7feecb4aa700 1 --2- 172.21.15.146:0/3098352081 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feec406b960 0x7feec4069e60 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7feec0009fb0 tx=0x7feec0009b30).stop 2024-01-21T20:05:01.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:01.042+0000 7feecb4aa700 1 -- 172.21.15.146:0/3098352081 shutdown_connections 2024-01-21T20:05:01.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:01.042+0000 7feecb4aa700 1 --2- 172.21.15.146:0/3098352081 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feec406b960 0x7feec4069e60 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:01.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:01.042+0000 7feecb4aa700 1 -- 172.21.15.146:0/3098352081 >> 172.21.15.146:0/3098352081 conn(0x7feec40f1c70 msgr2=0x7feec40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:01.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:01.042+0000 7feecb4aa700 1 -- 172.21.15.146:0/3098352081 shutdown_connections 2024-01-21T20:05:01.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:01.042+0000 7feecb4aa700 1 -- 172.21.15.146:0/3098352081 wait complete. 2024-01-21T20:05:01.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:01.042+0000 7feecb4aa700 1 Processor -- start 2024-01-21T20:05:01.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:01.042+0000 7feecb4aa700 1 -- start start 2024-01-21T20:05:01.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:01.042+0000 7feecb4aa700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feec406b960 0x7feec41088c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:01.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:01.042+0000 7feecb4aa700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feec4108db0 con 0x7feec406b960 2024-01-21T20:05:01.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:01.043+0000 7feec9246700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feec406b960 0x7feec41088c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:01.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:01.043+0000 7feec9246700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feec406b960 0x7feec41088c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:40992/0 (socket says 172.21.15.146:40992) 2024-01-21T20:05:01.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:01.043+0000 7feec9246700 1 -- 172.21.15.146:0/2514361308 learned_addr learned my addr 172.21.15.146:0/2514361308 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:05:01.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:01.043+0000 7feec9246700 1 -- 172.21.15.146:0/2514361308 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feec0005040 con 0x7feec406b960 2024-01-21T20:05:01.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:01.043+0000 7feec9246700 1 --2- 172.21.15.146:0/2514361308 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feec406b960 0x7feec41088c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7feec0013510 tx=0x7feec000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:01.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:01.043+0000 7feeb9ffb700 1 -- 172.21.15.146:0/2514361308 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feec0009320 con 0x7feec406b960 2024-01-21T20:05:01.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:01.043+0000 7feecb4aa700 1 -- 172.21.15.146:0/2514361308 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feec4108fb0 con 0x7feec406b960 2024-01-21T20:05:01.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:01.043+0000 7feeb9ffb700 1 -- 172.21.15.146:0/2514361308 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feec0013f00 con 0x7feec406b960 2024-01-21T20:05:01.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:01.043+0000 7feeb9ffb700 1 -- 172.21.15.146:0/2514361308 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feec0014dc0 con 0x7feec406b960 2024-01-21T20:05:01.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:01.043+0000 7feecb4aa700 1 -- 172.21.15.146:0/2514361308 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feec4109440 con 0x7feec406b960 2024-01-21T20:05:01.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:01.044+0000 7feeb9ffb700 1 -- 172.21.15.146:0/2514361308 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feec0014420 con 0x7feec406b960 2024-01-21T20:05:01.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:01.044+0000 7feecb4aa700 1 -- 172.21.15.146:0/2514361308 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feec4045ad0 con 0x7feec406b960 2024-01-21T20:05:01.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:01.045+0000 7feeb9ffb700 1 --2- 172.21.15.146:0/2514361308 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7feeb002ff60 0x7feeb0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:01.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:01.045+0000 7feeb9ffb700 1 -- 172.21.15.146:0/2514361308 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feec0041020 con 0x7feec406b960 2024-01-21T20:05:01.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:01.045+0000 7feec8a45700 1 --2- 172.21.15.146:0/2514361308 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7feeb002ff60 0x7feeb0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:01.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:01.045+0000 7feec8a45700 1 --2- 172.21.15.146:0/2514361308 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7feeb002ff60 0x7feeb0032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7feeb40096a0 tx=0x7feeb4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:01.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:01.049+0000 7feeb9ffb700 1 -- 172.21.15.146:0/2514361308 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feec00095f0 con 0x7feec406b960 2024-01-21T20:05:01.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:00 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:04:59.180454+0000 mgr.smithi053.fgsmol (mgr.14164) 181 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:01.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:01.252+0000 7feecb4aa700 1 -- 172.21.15.146:0/2514361308 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feec40fe150 con 0x7feec406b960 2024-01-21T20:05:01.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:01.253+0000 7feeb9ffb700 1 -- 172.21.15.146:0/2514361308 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7feec001cb10 con 0x7feec406b960 2024-01-21T20:05:01.244 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:05:01.244 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:05:01.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:01.255+0000 7feecb4aa700 1 -- 172.21.15.146:0/2514361308 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7feeb002ff60 msgr2=0x7feeb0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:01.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:01.255+0000 7feecb4aa700 1 --2- 172.21.15.146:0/2514361308 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7feeb002ff60 0x7feeb0032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7feeb40096a0 tx=0x7feeb4006b40).stop 2024-01-21T20:05:01.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:01.255+0000 7feecb4aa700 1 -- 172.21.15.146:0/2514361308 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feec406b960 msgr2=0x7feec41088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:01.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:01.255+0000 7feecb4aa700 1 --2- 172.21.15.146:0/2514361308 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feec406b960 0x7feec41088c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7feec0013510 tx=0x7feec000b040).stop 2024-01-21T20:05:01.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:01.255+0000 7feecb4aa700 1 -- 172.21.15.146:0/2514361308 shutdown_connections 2024-01-21T20:05:01.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:01.255+0000 7feecb4aa700 1 --2- 172.21.15.146:0/2514361308 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7feeb002ff60 0x7feeb0032420 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:01.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:01.255+0000 7feecb4aa700 1 --2- 172.21.15.146:0/2514361308 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feec406b960 0x7feec41088c0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:01.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:01.255+0000 7feecb4aa700 1 -- 172.21.15.146:0/2514361308 >> 172.21.15.146:0/2514361308 conn(0x7feec40f1c70 msgr2=0x7feec40f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:01.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:01.255+0000 7feecb4aa700 1 -- 172.21.15.146:0/2514361308 shutdown_connections 2024-01-21T20:05:01.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:01.255+0000 7feecb4aa700 1 -- 172.21.15.146:0/2514361308 wait complete. 2024-01-21T20:05:01.247 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:05:02.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:01 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:05:01.254437+0000 mon.smithi053 (mon.0) 249 : audit [DBG] from='client.? 172.21.15.146:0/2514361308' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:05:02.675 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:05:02.676 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:05:03.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:02 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:05:01.180704+0000 mgr.smithi053.fgsmol (mgr.14164) 182 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:03.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:03.891+0000 7feeccd27700 1 -- 172.21.15.146:0/2744211045 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feec80601e0 msgr2=0x7feec80f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:03.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:03.891+0000 7feeccd27700 1 --2- 172.21.15.146:0/2744211045 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feec80601e0 0x7feec80f8f20 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7feebc009fb0 tx=0x7feebc009b30).stop 2024-01-21T20:05:03.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:03.891+0000 7feeccd27700 1 -- 172.21.15.146:0/2744211045 shutdown_connections 2024-01-21T20:05:03.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:03.891+0000 7feeccd27700 1 --2- 172.21.15.146:0/2744211045 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feec80601e0 0x7feec80f8f20 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:03.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:03.891+0000 7feeccd27700 1 -- 172.21.15.146:0/2744211045 >> 172.21.15.146:0/2744211045 conn(0x7feec80f09a0 msgr2=0x7feec80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:03.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:03.891+0000 7feeccd27700 1 -- 172.21.15.146:0/2744211045 shutdown_connections 2024-01-21T20:05:03.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:03.891+0000 7feeccd27700 1 -- 172.21.15.146:0/2744211045 wait complete. 2024-01-21T20:05:03.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:03.891+0000 7feeccd27700 1 Processor -- start 2024-01-21T20:05:03.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:03.892+0000 7feeccd27700 1 -- start start 2024-01-21T20:05:03.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:03.892+0000 7feeccd27700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feec80601e0 0x7feec81064f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:03.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:03.892+0000 7feeccd27700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feec81069e0 con 0x7feec80601e0 2024-01-21T20:05:03.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:03.892+0000 7feec77fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feec80601e0 0x7feec81064f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:03.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:03.892+0000 7feec77fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feec80601e0 0x7feec81064f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:41010/0 (socket says 172.21.15.146:41010) 2024-01-21T20:05:03.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:03.892+0000 7feec77fe700 1 -- 172.21.15.146:0/1682663500 learned_addr learned my addr 172.21.15.146:0/1682663500 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:05:03.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:03.893+0000 7feec77fe700 1 -- 172.21.15.146:0/1682663500 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feebc005040 con 0x7feec80601e0 2024-01-21T20:05:03.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:03.893+0000 7feec77fe700 1 --2- 172.21.15.146:0/1682663500 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feec80601e0 0x7feec81064f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7feebc013510 tx=0x7feebc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:03.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:03.893+0000 7feeaffff700 1 -- 172.21.15.146:0/1682663500 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feebc009320 con 0x7feec80601e0 2024-01-21T20:05:03.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:03.893+0000 7feeaffff700 1 -- 172.21.15.146:0/1682663500 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feebc013f00 con 0x7feec80601e0 2024-01-21T20:05:03.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:03.893+0000 7feeccd27700 1 -- 172.21.15.146:0/1682663500 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feec8106be0 con 0x7feec80601e0 2024-01-21T20:05:03.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:03.893+0000 7feeaffff700 1 -- 172.21.15.146:0/1682663500 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feebc014d90 con 0x7feec80601e0 2024-01-21T20:05:03.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:03.893+0000 7feeccd27700 1 -- 172.21.15.146:0/1682663500 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feec8107070 con 0x7feec80601e0 2024-01-21T20:05:03.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:03.894+0000 7feeaffff700 1 -- 172.21.15.146:0/1682663500 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feebc0095f0 con 0x7feec80601e0 2024-01-21T20:05:03.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:03.894+0000 7feeccd27700 1 -- 172.21.15.146:0/1682663500 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feec8045ad0 con 0x7feec80601e0 2024-01-21T20:05:03.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:03.894+0000 7feeaffff700 1 --2- 172.21.15.146:0/1682663500 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7feeb002ff60 0x7feeb0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:03.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:03.894+0000 7feeaffff700 1 -- 172.21.15.146:0/1682663500 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feebc041000 con 0x7feec80601e0 2024-01-21T20:05:03.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:03.894+0000 7feec6ffd700 1 --2- 172.21.15.146:0/1682663500 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7feeb002ff60 0x7feeb0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:03.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:03.894+0000 7feec6ffd700 1 --2- 172.21.15.146:0/1682663500 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7feeb002ff60 0x7feeb0032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7feeb80096a0 tx=0x7feeb8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:03.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:03.899+0000 7feeaffff700 1 -- 172.21.15.146:0/1682663500 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feebc01d7f0 con 0x7feec80601e0 2024-01-21T20:05:04.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:04.102+0000 7feeccd27700 1 -- 172.21.15.146:0/1682663500 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feec8107f20 con 0x7feec80601e0 2024-01-21T20:05:04.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:04.103+0000 7feeaffff700 1 -- 172.21.15.146:0/1682663500 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7feebc010230 con 0x7feec80601e0 2024-01-21T20:05:04.094 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:05:04.094 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:05:04.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:04.105+0000 7feeccd27700 1 -- 172.21.15.146:0/1682663500 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7feeb002ff60 msgr2=0x7feeb0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:04.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:04.105+0000 7feeccd27700 1 --2- 172.21.15.146:0/1682663500 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7feeb002ff60 0x7feeb0032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7feeb80096a0 tx=0x7feeb8006b40).stop 2024-01-21T20:05:04.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:04.105+0000 7feeccd27700 1 -- 172.21.15.146:0/1682663500 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feec80601e0 msgr2=0x7feec81064f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:04.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:04.105+0000 7feeccd27700 1 --2- 172.21.15.146:0/1682663500 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feec80601e0 0x7feec81064f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7feebc013510 tx=0x7feebc00b040).stop 2024-01-21T20:05:04.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:04.105+0000 7feeccd27700 1 -- 172.21.15.146:0/1682663500 shutdown_connections 2024-01-21T20:05:04.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:04.105+0000 7feeccd27700 1 --2- 172.21.15.146:0/1682663500 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7feeb002ff60 0x7feeb0032420 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:04.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:04.105+0000 7feeccd27700 1 --2- 172.21.15.146:0/1682663500 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feec80601e0 0x7feec81064f0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:04.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:04.105+0000 7feeccd27700 1 -- 172.21.15.146:0/1682663500 >> 172.21.15.146:0/1682663500 conn(0x7feec80f09a0 msgr2=0x7feec80f1430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:04.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:04.105+0000 7feeccd27700 1 -- 172.21.15.146:0/1682663500 shutdown_connections 2024-01-21T20:05:04.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:04.105+0000 7feeccd27700 1 -- 172.21.15.146:0/1682663500 wait complete. 2024-01-21T20:05:04.097 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:05:05.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:04 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:05:03.181050+0000 mgr.smithi053.fgsmol (mgr.14164) 183 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:05.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:04 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:05:04.104530+0000 mon.smithi053 (mon.0) 250 : audit [DBG] from='client.? 172.21.15.146:0/1682663500' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:05:05.522 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:05:05.522 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:05:06.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:06.744+0000 7ff2f359e700 1 -- 172.21.15.146:0/2652430228 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2f4060280 msgr2=0x7ff2f40f8e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:06.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:06.744+0000 7ff2f359e700 1 --2- 172.21.15.146:0/2652430228 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2f4060280 0x7ff2f40f8e20 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7ff2e8009fb0 tx=0x7ff2e8009b30).stop 2024-01-21T20:05:06.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:06.744+0000 7ff2f359e700 1 -- 172.21.15.146:0/2652430228 shutdown_connections 2024-01-21T20:05:06.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:06.744+0000 7ff2f359e700 1 --2- 172.21.15.146:0/2652430228 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2f4060280 0x7ff2f40f8e20 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:06.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:06.744+0000 7ff2f359e700 1 -- 172.21.15.146:0/2652430228 >> 172.21.15.146:0/2652430228 conn(0x7ff2f40f08c0 msgr2=0x7ff2f40f2ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:06.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:06.744+0000 7ff2f359e700 1 -- 172.21.15.146:0/2652430228 shutdown_connections 2024-01-21T20:05:06.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:06.744+0000 7ff2f359e700 1 -- 172.21.15.146:0/2652430228 wait complete. 2024-01-21T20:05:06.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:06.744+0000 7ff2f359e700 1 Processor -- start 2024-01-21T20:05:06.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:06.744+0000 7ff2f359e700 1 -- start start 2024-01-21T20:05:06.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:06.744+0000 7ff2f359e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2f4060280 0x7ff2f4106460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:06.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:06.744+0000 7ff2f359e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f4106950 con 0x7ff2f4060280 2024-01-21T20:05:06.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:06.745+0000 7ff2f259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2f4060280 0x7ff2f4106460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:06.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:06.745+0000 7ff2f259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2f4060280 0x7ff2f4106460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:41032/0 (socket says 172.21.15.146:41032) 2024-01-21T20:05:06.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:06.745+0000 7ff2f259c700 1 -- 172.21.15.146:0/2006943009 learned_addr learned my addr 172.21.15.146:0/2006943009 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:05:06.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:06.745+0000 7ff2f259c700 1 -- 172.21.15.146:0/2006943009 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e8005040 con 0x7ff2f4060280 2024-01-21T20:05:06.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:06.745+0000 7ff2f259c700 1 --2- 172.21.15.146:0/2006943009 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2f4060280 0x7ff2f4106460 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7ff2e8013510 tx=0x7ff2e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:06.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:06.745+0000 7ff2e2ffd700 1 -- 172.21.15.146:0/2006943009 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2e8009320 con 0x7ff2f4060280 2024-01-21T20:05:06.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:06.745+0000 7ff2e2ffd700 1 -- 172.21.15.146:0/2006943009 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff2e8013f00 con 0x7ff2f4060280 2024-01-21T20:05:06.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:06.745+0000 7ff2f359e700 1 -- 172.21.15.146:0/2006943009 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2f4106b50 con 0x7ff2f4060280 2024-01-21T20:05:06.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:06.745+0000 7ff2e2ffd700 1 -- 172.21.15.146:0/2006943009 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2e8014dc0 con 0x7ff2f4060280 2024-01-21T20:05:06.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:06.745+0000 7ff2f359e700 1 -- 172.21.15.146:0/2006943009 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2f4106fe0 con 0x7ff2f4060280 2024-01-21T20:05:06.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:06.746+0000 7ff2e2ffd700 1 -- 172.21.15.146:0/2006943009 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff2e80095f0 con 0x7ff2f4060280 2024-01-21T20:05:06.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:06.746+0000 7ff2f359e700 1 -- 172.21.15.146:0/2006943009 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2f4045ad0 con 0x7ff2f4060280 2024-01-21T20:05:06.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:06.746+0000 7ff2e2ffd700 1 --2- 172.21.15.146:0/2006943009 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff2dc02ff60 0x7ff2dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:06.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:06.746+0000 7ff2e2ffd700 1 -- 172.21.15.146:0/2006943009 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff2e80410c0 con 0x7ff2f4060280 2024-01-21T20:05:06.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:06.746+0000 7ff2f1d9b700 1 --2- 172.21.15.146:0/2006943009 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff2dc02ff60 0x7ff2dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:06.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:06.747+0000 7ff2f1d9b700 1 --2- 172.21.15.146:0/2006943009 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff2dc02ff60 0x7ff2dc032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7ff2e40096a0 tx=0x7ff2e4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:06.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:06.751+0000 7ff2e2ffd700 1 -- 172.21.15.146:0/2006943009 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff2e801c790 con 0x7ff2f4060280 2024-01-21T20:05:06.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:06.955+0000 7ff2f359e700 1 -- 172.21.15.146:0/2006943009 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff2f4107e90 con 0x7ff2f4060280 2024-01-21T20:05:06.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:06.956+0000 7ff2e2ffd700 1 -- 172.21.15.146:0/2006943009 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff2e8010230 con 0x7ff2f4060280 2024-01-21T20:05:06.947 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:05:06.947 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:05:06.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:06.957+0000 7ff2f359e700 1 -- 172.21.15.146:0/2006943009 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff2dc02ff60 msgr2=0x7ff2dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:06.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:06.957+0000 7ff2f359e700 1 --2- 172.21.15.146:0/2006943009 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff2dc02ff60 0x7ff2dc032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7ff2e40096a0 tx=0x7ff2e4006b40).stop 2024-01-21T20:05:06.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:06.957+0000 7ff2f359e700 1 -- 172.21.15.146:0/2006943009 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2f4060280 msgr2=0x7ff2f4106460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:06.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:06.957+0000 7ff2f359e700 1 --2- 172.21.15.146:0/2006943009 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2f4060280 0x7ff2f4106460 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7ff2e8013510 tx=0x7ff2e800b040).stop 2024-01-21T20:05:06.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:06.957+0000 7ff2f359e700 1 -- 172.21.15.146:0/2006943009 shutdown_connections 2024-01-21T20:05:06.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:06.957+0000 7ff2f359e700 1 --2- 172.21.15.146:0/2006943009 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff2dc02ff60 0x7ff2dc032420 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:06.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:06.957+0000 7ff2f359e700 1 --2- 172.21.15.146:0/2006943009 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2f4060280 0x7ff2f4106460 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:06.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:06.957+0000 7ff2f359e700 1 -- 172.21.15.146:0/2006943009 >> 172.21.15.146:0/2006943009 conn(0x7ff2f40f08c0 msgr2=0x7ff2f40f1380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:06.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:06.957+0000 7ff2f359e700 1 -- 172.21.15.146:0/2006943009 shutdown_connections 2024-01-21T20:05:06.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:06.957+0000 7ff2f359e700 1 -- 172.21.15.146:0/2006943009 wait complete. 2024-01-21T20:05:06.949 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:05:07.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:06 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:05:05.181376+0000 mgr.smithi053.fgsmol (mgr.14164) 184 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:08.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:07 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:05:06.957124+0000 mon.smithi053 (mon.0) 251 : audit [DBG] from='client.? 172.21.15.146:0/2006943009' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:05:08.368 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:05:08.368 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:05:09.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:08 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:05:07.181698+0000 mgr.smithi053.fgsmol (mgr.14164) 185 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:09.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:09.546+0000 7f0ac6708700 1 -- 172.21.15.146:0/3500920226 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0ac00f5410 msgr2=0x7f0ac00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:09.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:09.546+0000 7f0ac6708700 1 --2- 172.21.15.146:0/3500920226 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0ac00f5410 0x7f0ac00f5830 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f0ab0009fb0 tx=0x7f0ab0009b30).stop 2024-01-21T20:05:09.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:09.546+0000 7f0ac6708700 1 -- 172.21.15.146:0/3500920226 shutdown_connections 2024-01-21T20:05:09.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:09.546+0000 7f0ac6708700 1 --2- 172.21.15.146:0/3500920226 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0ac00f5410 0x7f0ac00f5830 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:09.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:09.546+0000 7f0ac6708700 1 -- 172.21.15.146:0/3500920226 >> 172.21.15.146:0/3500920226 conn(0x7f0ac00f0b70 msgr2=0x7f0ac00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:09.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:09.546+0000 7f0ac6708700 1 -- 172.21.15.146:0/3500920226 shutdown_connections 2024-01-21T20:05:09.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:09.546+0000 7f0ac6708700 1 -- 172.21.15.146:0/3500920226 wait complete. 2024-01-21T20:05:09.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:09.547+0000 7f0ac6708700 1 Processor -- start 2024-01-21T20:05:09.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:09.547+0000 7f0ac6708700 1 -- start start 2024-01-21T20:05:09.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:09.547+0000 7f0ac6708700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0ac00f5410 0x7f0ac0105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:09.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:09.547+0000 7f0ac6708700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac0105af0 con 0x7f0ac00f5410 2024-01-21T20:05:09.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:09.548+0000 7f0abffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0ac00f5410 0x7f0ac0105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:09.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:09.548+0000 7f0abffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0ac00f5410 0x7f0ac0105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:43418/0 (socket says 172.21.15.146:43418) 2024-01-21T20:05:09.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:09.548+0000 7f0abffff700 1 -- 172.21.15.146:0/3891684065 learned_addr learned my addr 172.21.15.146:0/3891684065 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:05:09.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:09.548+0000 7f0abffff700 1 -- 172.21.15.146:0/3891684065 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab0005040 con 0x7f0ac00f5410 2024-01-21T20:05:09.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:09.548+0000 7f0abffff700 1 --2- 172.21.15.146:0/3891684065 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0ac00f5410 0x7f0ac0105600 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f0ab0013510 tx=0x7f0ab000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:09.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:09.548+0000 7f0abcff9700 1 -- 172.21.15.146:0/3891684065 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ab0009320 con 0x7f0ac00f5410 2024-01-21T20:05:09.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:09.549+0000 7f0ac6708700 1 -- 172.21.15.146:0/3891684065 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ac0105cf0 con 0x7f0ac00f5410 2024-01-21T20:05:09.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:09.549+0000 7f0abcff9700 1 -- 172.21.15.146:0/3891684065 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0ab0013f00 con 0x7f0ac00f5410 2024-01-21T20:05:09.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:09.549+0000 7f0abcff9700 1 -- 172.21.15.146:0/3891684065 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ab0014d90 con 0x7f0ac00f5410 2024-01-21T20:05:09.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:09.549+0000 7f0ac6708700 1 -- 172.21.15.146:0/3891684065 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ac0106180 con 0x7f0ac00f5410 2024-01-21T20:05:09.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:09.549+0000 7f0abcff9700 1 -- 172.21.15.146:0/3891684065 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0ab00095f0 con 0x7f0ac00f5410 2024-01-21T20:05:09.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:09.549+0000 7f0ac6708700 1 -- 172.21.15.146:0/3891684065 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ac0045ad0 con 0x7f0ac00f5410 2024-01-21T20:05:09.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:09.550+0000 7f0abcff9700 1 --2- 172.21.15.146:0/3891684065 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0aa802ff60 0x7f0aa8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:09.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:09.550+0000 7f0abcff9700 1 -- 172.21.15.146:0/3891684065 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0ab0040fe0 con 0x7f0ac00f5410 2024-01-21T20:05:09.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:09.550+0000 7f0abf7fe700 1 --2- 172.21.15.146:0/3891684065 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0aa802ff60 0x7f0aa8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:09.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:09.550+0000 7f0abf7fe700 1 --2- 172.21.15.146:0/3891684065 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0aa802ff60 0x7f0aa8032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f0ab40096a0 tx=0x7f0ab4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:09.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:09.555+0000 7f0abcff9700 1 -- 172.21.15.146:0/3891684065 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0ab001d7f0 con 0x7f0ac00f5410 2024-01-21T20:05:09.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:09.760+0000 7f0ac6708700 1 -- 172.21.15.146:0/3891684065 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0ac00fae60 con 0x7f0ac00f5410 2024-01-21T20:05:09.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:09.760+0000 7f0abcff9700 1 -- 172.21.15.146:0/3891684065 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0ab0010230 con 0x7f0ac00f5410 2024-01-21T20:05:09.751 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:05:09.751 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:05:09.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:09.762+0000 7f0ac6708700 1 -- 172.21.15.146:0/3891684065 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0aa802ff60 msgr2=0x7f0aa8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:09.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:09.762+0000 7f0ac6708700 1 --2- 172.21.15.146:0/3891684065 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0aa802ff60 0x7f0aa8032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f0ab40096a0 tx=0x7f0ab4006b40).stop 2024-01-21T20:05:09.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:09.762+0000 7f0ac6708700 1 -- 172.21.15.146:0/3891684065 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0ac00f5410 msgr2=0x7f0ac0105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:09.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:09.762+0000 7f0ac6708700 1 --2- 172.21.15.146:0/3891684065 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0ac00f5410 0x7f0ac0105600 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f0ab0013510 tx=0x7f0ab000b040).stop 2024-01-21T20:05:09.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:09.762+0000 7f0ac6708700 1 -- 172.21.15.146:0/3891684065 shutdown_connections 2024-01-21T20:05:09.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:09.762+0000 7f0ac6708700 1 --2- 172.21.15.146:0/3891684065 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0aa802ff60 0x7f0aa8032420 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:09.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:09.762+0000 7f0ac6708700 1 --2- 172.21.15.146:0/3891684065 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0ac00f5410 0x7f0ac0105600 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:09.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:09.762+0000 7f0ac6708700 1 -- 172.21.15.146:0/3891684065 >> 172.21.15.146:0/3891684065 conn(0x7f0ac00f0b70 msgr2=0x7f0ac00f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:09.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:09.762+0000 7f0ac6708700 1 -- 172.21.15.146:0/3891684065 shutdown_connections 2024-01-21T20:05:09.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:09.762+0000 7f0ac6708700 1 -- 172.21.15.146:0/3891684065 wait complete. 2024-01-21T20:05:09.754 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:05:10.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:09 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:05:09.268648+0000 mon.smithi053 (mon.0) 252 : audit [INF] from='mgr.14164 172.21.15.53:0/2331671680' entity='mgr.smithi053.fgsmol' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fgsmol/mirror_snapshot_schedule"}]: dispatch 2024-01-21T20:05:10.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:09 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:05:09.276895+0000 mon.smithi053 (mon.0) 253 : audit [INF] from='mgr.14164 172.21.15.53:0/2331671680' entity='mgr.smithi053.fgsmol' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fgsmol/trash_purge_schedule"}]: dispatch 2024-01-21T20:05:10.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:09 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:05:09.761718+0000 mon.smithi053 (mon.0) 254 : audit [DBG] from='client.? 172.21.15.146:0/3891684065' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:05:11.170 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:05:11.171 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:05:11.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:10 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:05:09.182053+0000 mgr.smithi053.fgsmol (mgr.14164) 186 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:12.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:12.484+0000 7f2d4a914700 1 -- 172.21.15.146:0/1673190604 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2d440f6510 msgr2=0x7f2d440f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:12.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:12.484+0000 7f2d4a914700 1 --2- 172.21.15.146:0/1673190604 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2d440f6510 0x7f2d440f6930 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f2d34009fb0 tx=0x7f2d34009b30).stop 2024-01-21T20:05:12.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:12.484+0000 7f2d4a914700 1 -- 172.21.15.146:0/1673190604 shutdown_connections 2024-01-21T20:05:12.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:12.484+0000 7f2d4a914700 1 --2- 172.21.15.146:0/1673190604 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2d440f6510 0x7f2d440f6930 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:12.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:12.484+0000 7f2d4a914700 1 -- 172.21.15.146:0/1673190604 >> 172.21.15.146:0/1673190604 conn(0x7f2d440f1c70 msgr2=0x7f2d440f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:12.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:12.484+0000 7f2d4a914700 1 -- 172.21.15.146:0/1673190604 shutdown_connections 2024-01-21T20:05:12.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:12.484+0000 7f2d4a914700 1 -- 172.21.15.146:0/1673190604 wait complete. 2024-01-21T20:05:12.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:12.485+0000 7f2d4a914700 1 Processor -- start 2024-01-21T20:05:12.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:12.485+0000 7f2d4a914700 1 -- start start 2024-01-21T20:05:12.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:12.485+0000 7f2d4a914700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2d440f6510 0x7f2d44106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:12.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:12.485+0000 7f2d4a914700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d44106c20 con 0x7f2d440f6510 2024-01-21T20:05:12.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:12.485+0000 7f2d43fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2d440f6510 0x7f2d44106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:12.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:12.485+0000 7f2d43fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2d440f6510 0x7f2d44106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:43440/0 (socket says 172.21.15.146:43440) 2024-01-21T20:05:12.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:12.485+0000 7f2d43fff700 1 -- 172.21.15.146:0/4022385112 learned_addr learned my addr 172.21.15.146:0/4022385112 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:05:12.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:12.486+0000 7f2d43fff700 1 -- 172.21.15.146:0/4022385112 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d34005040 con 0x7f2d440f6510 2024-01-21T20:05:12.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:12.486+0000 7f2d43fff700 1 --2- 172.21.15.146:0/4022385112 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2d440f6510 0x7f2d44106730 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f2d34013510 tx=0x7f2d3400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:12.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:12.486+0000 7f2d40ff9700 1 -- 172.21.15.146:0/4022385112 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d34009320 con 0x7f2d440f6510 2024-01-21T20:05:12.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:12.486+0000 7f2d40ff9700 1 -- 172.21.15.146:0/4022385112 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2d34013f00 con 0x7f2d440f6510 2024-01-21T20:05:12.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:12.486+0000 7f2d4a914700 1 -- 172.21.15.146:0/4022385112 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d44106e20 con 0x7f2d440f6510 2024-01-21T20:05:12.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:12.486+0000 7f2d40ff9700 1 -- 172.21.15.146:0/4022385112 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d34014d90 con 0x7f2d440f6510 2024-01-21T20:05:12.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:12.486+0000 7f2d4a914700 1 -- 172.21.15.146:0/4022385112 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d441072b0 con 0x7f2d440f6510 2024-01-21T20:05:12.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:12.487+0000 7f2d40ff9700 1 -- 172.21.15.146:0/4022385112 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2d340095f0 con 0x7f2d440f6510 2024-01-21T20:05:12.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:12.487+0000 7f2d4a914700 1 -- 172.21.15.146:0/4022385112 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d44045ad0 con 0x7f2d440f6510 2024-01-21T20:05:12.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:12.487+0000 7f2d40ff9700 1 --2- 172.21.15.146:0/4022385112 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f2d2c02ff60 0x7f2d2c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:12.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:12.487+0000 7f2d40ff9700 1 -- 172.21.15.146:0/4022385112 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2d34041080 con 0x7f2d440f6510 2024-01-21T20:05:12.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:12.487+0000 7f2d437fe700 1 --2- 172.21.15.146:0/4022385112 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f2d2c02ff60 0x7f2d2c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:12.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:12.488+0000 7f2d437fe700 1 --2- 172.21.15.146:0/4022385112 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f2d2c02ff60 0x7f2d2c032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f2d380096a0 tx=0x7f2d38006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:12.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:12.492+0000 7f2d40ff9700 1 -- 172.21.15.146:0/4022385112 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2d34014420 con 0x7f2d440f6510 2024-01-21T20:05:12.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:12.696+0000 7f2d4a914700 1 -- 172.21.15.146:0/4022385112 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2d440fbf60 con 0x7f2d440f6510 2024-01-21T20:05:12.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:12.696+0000 7f2d40ff9700 1 -- 172.21.15.146:0/4022385112 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2d3401d7f0 con 0x7f2d440f6510 2024-01-21T20:05:12.687 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:05:12.688 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:05:12.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:12.698+0000 7f2d4a914700 1 -- 172.21.15.146:0/4022385112 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f2d2c02ff60 msgr2=0x7f2d2c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:12.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:12.698+0000 7f2d4a914700 1 --2- 172.21.15.146:0/4022385112 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f2d2c02ff60 0x7f2d2c032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f2d380096a0 tx=0x7f2d38006b40).stop 2024-01-21T20:05:12.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:12.698+0000 7f2d4a914700 1 -- 172.21.15.146:0/4022385112 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2d440f6510 msgr2=0x7f2d44106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:12.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:12.698+0000 7f2d4a914700 1 --2- 172.21.15.146:0/4022385112 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2d440f6510 0x7f2d44106730 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f2d34013510 tx=0x7f2d3400b040).stop 2024-01-21T20:05:12.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:12.698+0000 7f2d4a914700 1 -- 172.21.15.146:0/4022385112 shutdown_connections 2024-01-21T20:05:12.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:12.698+0000 7f2d4a914700 1 --2- 172.21.15.146:0/4022385112 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f2d2c02ff60 0x7f2d2c032420 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:12.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:12.698+0000 7f2d4a914700 1 --2- 172.21.15.146:0/4022385112 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2d440f6510 0x7f2d44106730 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:12.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:12.698+0000 7f2d4a914700 1 -- 172.21.15.146:0/4022385112 >> 172.21.15.146:0/4022385112 conn(0x7f2d440f1c70 msgr2=0x7f2d440f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:12.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:12.698+0000 7f2d4a914700 1 -- 172.21.15.146:0/4022385112 shutdown_connections 2024-01-21T20:05:12.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:12.698+0000 7f2d4a914700 1 -- 172.21.15.146:0/4022385112 wait complete. 2024-01-21T20:05:12.690 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:05:13.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:12 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:05:11.182438+0000 mgr.smithi053.fgsmol (mgr.14164) 187 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:13.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:12 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:05:12.697832+0000 mon.smithi053 (mon.0) 255 : audit [DBG] from='client.? 172.21.15.146:0/4022385112' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:05:14.107 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:05:14.108 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:05:15.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:14 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:05:13.182656+0000 mgr.smithi053.fgsmol (mgr.14164) 188 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:15.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:15.361+0000 7f9ecffff700 1 -- 172.21.15.146:0/3795836085 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ed00f51a0 msgr2=0x7f9ed00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:15.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:15.361+0000 7f9ecffff700 1 --2- 172.21.15.146:0/3795836085 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ed00f51a0 0x7f9ed00f55c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f9ec4009fb0 tx=0x7f9ec4009b30).stop 2024-01-21T20:05:15.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:15.361+0000 7f9ecffff700 1 -- 172.21.15.146:0/3795836085 shutdown_connections 2024-01-21T20:05:15.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:15.361+0000 7f9ecffff700 1 --2- 172.21.15.146:0/3795836085 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ed00f51a0 0x7f9ed00f55c0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:15.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:15.361+0000 7f9ecffff700 1 -- 172.21.15.146:0/3795836085 >> 172.21.15.146:0/3795836085 conn(0x7f9ed00f09a0 msgr2=0x7f9ed00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:15.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:15.361+0000 7f9ecffff700 1 -- 172.21.15.146:0/3795836085 shutdown_connections 2024-01-21T20:05:15.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:15.361+0000 7f9ecffff700 1 -- 172.21.15.146:0/3795836085 wait complete. 2024-01-21T20:05:15.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:15.362+0000 7f9ecffff700 1 Processor -- start 2024-01-21T20:05:15.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:15.362+0000 7f9ecffff700 1 -- start start 2024-01-21T20:05:15.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:15.362+0000 7f9ecffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ed00f51a0 0x7f9ed01064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:15.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:15.362+0000 7f9ecffff700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed0106990 con 0x7f9ed00f51a0 2024-01-21T20:05:15.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:15.362+0000 7f9eceffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ed00f51a0 0x7f9ed01064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:15.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:15.362+0000 7f9eceffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ed00f51a0 0x7f9ed01064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:43450/0 (socket says 172.21.15.146:43450) 2024-01-21T20:05:15.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:15.362+0000 7f9eceffd700 1 -- 172.21.15.146:0/3686630882 learned_addr learned my addr 172.21.15.146:0/3686630882 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:05:15.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:15.362+0000 7f9eceffd700 1 -- 172.21.15.146:0/3686630882 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec4005040 con 0x7f9ed00f51a0 2024-01-21T20:05:15.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:15.362+0000 7f9eceffd700 1 --2- 172.21.15.146:0/3686630882 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ed00f51a0 0x7f9ed01064a0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f9ec4013510 tx=0x7f9ec400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:15.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:15.363+0000 7f9eb77fe700 1 -- 172.21.15.146:0/3686630882 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ec4009320 con 0x7f9ed00f51a0 2024-01-21T20:05:15.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:15.363+0000 7f9ecffff700 1 -- 172.21.15.146:0/3686630882 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ed0106b90 con 0x7f9ed00f51a0 2024-01-21T20:05:15.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:15.363+0000 7f9eb77fe700 1 -- 172.21.15.146:0/3686630882 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9ec4013f00 con 0x7f9ed00f51a0 2024-01-21T20:05:15.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:15.363+0000 7f9eb77fe700 1 -- 172.21.15.146:0/3686630882 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ec4014dc0 con 0x7f9ed00f51a0 2024-01-21T20:05:15.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:15.363+0000 7f9ecffff700 1 -- 172.21.15.146:0/3686630882 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ed0107020 con 0x7f9ed00f51a0 2024-01-21T20:05:15.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:15.363+0000 7f9eb77fe700 1 -- 172.21.15.146:0/3686630882 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9ec401d3d0 con 0x7f9ed00f51a0 2024-01-21T20:05:15.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:15.364+0000 7f9eb77fe700 1 --2- 172.21.15.146:0/3686630882 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f9eb802ff60 0x7f9eb8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:15.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:15.364+0000 7f9eb77fe700 1 -- 172.21.15.146:0/3686630882 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9ec4041110 con 0x7f9ed00f51a0 2024-01-21T20:05:15.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:15.364+0000 7f9ecffff700 1 -- 172.21.15.146:0/3686630882 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ed0059100 con 0x7f9ed00f51a0 2024-01-21T20:05:15.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:15.364+0000 7f9ece7fc700 1 --2- 172.21.15.146:0/3686630882 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f9eb802ff60 0x7f9eb8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:15.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:15.364+0000 7f9ece7fc700 1 --2- 172.21.15.146:0/3686630882 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f9eb802ff60 0x7f9eb8032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f9ec00096a0 tx=0x7f9ec0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:15.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:15.369+0000 7f9eb77fe700 1 -- 172.21.15.146:0/3686630882 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9ec4012920 con 0x7f9ed00f51a0 2024-01-21T20:05:15.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:15.577+0000 7f9ecffff700 1 -- 172.21.15.146:0/3686630882 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9ed00fabe0 con 0x7f9ed00f51a0 2024-01-21T20:05:15.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:15.577+0000 7f9eb77fe700 1 -- 172.21.15.146:0/3686630882 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9ec402b020 con 0x7f9ed00f51a0 2024-01-21T20:05:15.568 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:05:15.569 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:05:15.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:15.579+0000 7f9ecffff700 1 -- 172.21.15.146:0/3686630882 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f9eb802ff60 msgr2=0x7f9eb8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:15.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:15.579+0000 7f9ecffff700 1 --2- 172.21.15.146:0/3686630882 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f9eb802ff60 0x7f9eb8032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f9ec00096a0 tx=0x7f9ec0006b40).stop 2024-01-21T20:05:15.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:15.579+0000 7f9ecffff700 1 -- 172.21.15.146:0/3686630882 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ed00f51a0 msgr2=0x7f9ed01064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:15.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:15.579+0000 7f9ecffff700 1 --2- 172.21.15.146:0/3686630882 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ed00f51a0 0x7f9ed01064a0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f9ec4013510 tx=0x7f9ec400b040).stop 2024-01-21T20:05:15.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:15.579+0000 7f9ecffff700 1 -- 172.21.15.146:0/3686630882 shutdown_connections 2024-01-21T20:05:15.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:15.579+0000 7f9ecffff700 1 --2- 172.21.15.146:0/3686630882 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f9eb802ff60 0x7f9eb8032420 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:15.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:15.579+0000 7f9ecffff700 1 --2- 172.21.15.146:0/3686630882 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ed00f51a0 0x7f9ed01064a0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:15.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:15.579+0000 7f9ecffff700 1 -- 172.21.15.146:0/3686630882 >> 172.21.15.146:0/3686630882 conn(0x7f9ed00f09a0 msgr2=0x7f9ed005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:15.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:15.579+0000 7f9ecffff700 1 -- 172.21.15.146:0/3686630882 shutdown_connections 2024-01-21T20:05:15.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:15.579+0000 7f9ecffff700 1 -- 172.21.15.146:0/3686630882 wait complete. 2024-01-21T20:05:15.571 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:05:16.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:15 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:05:15.578792+0000 mon.smithi053 (mon.0) 256 : audit [DBG] from='client.? 172.21.15.146:0/3686630882' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:05:16.979 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:05:16.979 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:05:17.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:05:15.182951+0000 mgr.smithi053.fgsmol (mgr.14164) 189 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:18.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:18.225+0000 7fc3289f9700 1 -- 172.21.15.146:0/1378667098 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc324060280 msgr2=0x7fc3240f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:18.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:18.225+0000 7fc3289f9700 1 --2- 172.21.15.146:0/1378667098 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc324060280 0x7fc3240f90f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7fc30c009fb0 tx=0x7fc30c009b30).stop 2024-01-21T20:05:18.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:18.226+0000 7fc3289f9700 1 -- 172.21.15.146:0/1378667098 shutdown_connections 2024-01-21T20:05:18.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:18.226+0000 7fc3289f9700 1 --2- 172.21.15.146:0/1378667098 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc324060280 0x7fc3240f90f0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:18.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:18.226+0000 7fc3289f9700 1 -- 172.21.15.146:0/1378667098 >> 172.21.15.146:0/1378667098 conn(0x7fc3240f0b70 msgr2=0x7fc3240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:18.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:18.226+0000 7fc3289f9700 1 -- 172.21.15.146:0/1378667098 shutdown_connections 2024-01-21T20:05:18.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:18.226+0000 7fc3289f9700 1 -- 172.21.15.146:0/1378667098 wait complete. 2024-01-21T20:05:18.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:18.226+0000 7fc3289f9700 1 Processor -- start 2024-01-21T20:05:18.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:18.226+0000 7fc3289f9700 1 -- start start 2024-01-21T20:05:18.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:18.227+0000 7fc3289f9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc324060280 0x7fc3241066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:18.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:18.227+0000 7fc3289f9700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc324106bd0 con 0x7fc324060280 2024-01-21T20:05:18.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:18.227+0000 7fc32259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc324060280 0x7fc3241066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:18.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:18.227+0000 7fc32259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc324060280 0x7fc3241066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:40216/0 (socket says 172.21.15.146:40216) 2024-01-21T20:05:18.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:18.227+0000 7fc32259c700 1 -- 172.21.15.146:0/4211456518 learned_addr learned my addr 172.21.15.146:0/4211456518 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:05:18.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:18.227+0000 7fc32259c700 1 -- 172.21.15.146:0/4211456518 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc30c005040 con 0x7fc324060280 2024-01-21T20:05:18.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:18.227+0000 7fc32259c700 1 --2- 172.21.15.146:0/4211456518 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc324060280 0x7fc3241066e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fc30c013510 tx=0x7fc30c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:18.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:18.227+0000 7fc31affd700 1 -- 172.21.15.146:0/4211456518 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc30c009320 con 0x7fc324060280 2024-01-21T20:05:18.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:18.227+0000 7fc3289f9700 1 -- 172.21.15.146:0/4211456518 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc324106dd0 con 0x7fc324060280 2024-01-21T20:05:18.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:18.227+0000 7fc31affd700 1 -- 172.21.15.146:0/4211456518 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc30c013f00 con 0x7fc324060280 2024-01-21T20:05:18.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:18.227+0000 7fc31affd700 1 -- 172.21.15.146:0/4211456518 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc30c014dc0 con 0x7fc324060280 2024-01-21T20:05:18.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:18.227+0000 7fc3289f9700 1 -- 172.21.15.146:0/4211456518 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc324107260 con 0x7fc324060280 2024-01-21T20:05:18.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:18.228+0000 7fc31affd700 1 -- 172.21.15.146:0/4211456518 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc30c01d3d0 con 0x7fc324060280 2024-01-21T20:05:18.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:18.228+0000 7fc31affd700 1 --2- 172.21.15.146:0/4211456518 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fc31002ffb0 0x7fc310032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:18.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:18.228+0000 7fc3289f9700 1 -- 172.21.15.146:0/4211456518 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc324045ad0 con 0x7fc324060280 2024-01-21T20:05:18.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:18.228+0000 7fc31affd700 1 -- 172.21.15.146:0/4211456518 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc30c0410f0 con 0x7fc324060280 2024-01-21T20:05:18.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:18.229+0000 7fc321d9b700 1 --2- 172.21.15.146:0/4211456518 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fc31002ffb0 0x7fc310032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:18.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:18.229+0000 7fc321d9b700 1 --2- 172.21.15.146:0/4211456518 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fc31002ffb0 0x7fc310032470 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fc3140096a0 tx=0x7fc314006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:18.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:18.234+0000 7fc31affd700 1 -- 172.21.15.146:0/4211456518 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc30c012920 con 0x7fc324060280 2024-01-21T20:05:18.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:18.439+0000 7fc3289f9700 1 -- 172.21.15.146:0/4211456518 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc324108110 con 0x7fc324060280 2024-01-21T20:05:18.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:18.439+0000 7fc31affd700 1 -- 172.21.15.146:0/4211456518 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc30c02b020 con 0x7fc324060280 2024-01-21T20:05:18.430 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:05:18.431 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:05:18.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:18.441+0000 7fc3289f9700 1 -- 172.21.15.146:0/4211456518 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fc31002ffb0 msgr2=0x7fc310032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:18.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:18.441+0000 7fc3289f9700 1 --2- 172.21.15.146:0/4211456518 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fc31002ffb0 0x7fc310032470 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fc3140096a0 tx=0x7fc314006b40).stop 2024-01-21T20:05:18.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:18.441+0000 7fc3289f9700 1 -- 172.21.15.146:0/4211456518 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc324060280 msgr2=0x7fc3241066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:18.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:18.441+0000 7fc3289f9700 1 --2- 172.21.15.146:0/4211456518 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc324060280 0x7fc3241066e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fc30c013510 tx=0x7fc30c00b040).stop 2024-01-21T20:05:18.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:18.441+0000 7fc3289f9700 1 -- 172.21.15.146:0/4211456518 shutdown_connections 2024-01-21T20:05:18.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:18.441+0000 7fc3289f9700 1 --2- 172.21.15.146:0/4211456518 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fc31002ffb0 0x7fc310032470 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:18.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:18.441+0000 7fc3289f9700 1 --2- 172.21.15.146:0/4211456518 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc324060280 0x7fc3241066e0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:18.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:18.441+0000 7fc3289f9700 1 -- 172.21.15.146:0/4211456518 >> 172.21.15.146:0/4211456518 conn(0x7fc3240f0b70 msgr2=0x7fc3240f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:18.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:18.441+0000 7fc3289f9700 1 -- 172.21.15.146:0/4211456518 shutdown_connections 2024-01-21T20:05:18.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:18.441+0000 7fc3289f9700 1 -- 172.21.15.146:0/4211456518 wait complete. 2024-01-21T20:05:18.433 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:05:19.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:18 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:05:17.183302+0000 mgr.smithi053.fgsmol (mgr.14164) 190 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:19.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:18 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:05:18.440716+0000 mon.smithi053 (mon.0) 257 : audit [DBG] from='client.? 172.21.15.146:0/4211456518' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:05:19.846 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:05:19.857 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:05:21.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:21.058+0000 7f4b4f944700 1 -- 172.21.15.146:0/505481205 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b480f5410 msgr2=0x7f4b480f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:21.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:21.058+0000 7f4b4f944700 1 --2- 172.21.15.146:0/505481205 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b480f5410 0x7f4b480f5830 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f4b44009fb0 tx=0x7f4b44009b30).stop 2024-01-21T20:05:21.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:21.058+0000 7f4b4f944700 1 -- 172.21.15.146:0/505481205 shutdown_connections 2024-01-21T20:05:21.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:21.058+0000 7f4b4f944700 1 --2- 172.21.15.146:0/505481205 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b480f5410 0x7f4b480f5830 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:21.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:21.058+0000 7f4b4f944700 1 -- 172.21.15.146:0/505481205 >> 172.21.15.146:0/505481205 conn(0x7f4b480f0b70 msgr2=0x7f4b480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:21.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:21.058+0000 7f4b4f944700 1 -- 172.21.15.146:0/505481205 shutdown_connections 2024-01-21T20:05:21.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:21.058+0000 7f4b4f944700 1 -- 172.21.15.146:0/505481205 wait complete. 2024-01-21T20:05:21.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:21.058+0000 7f4b4f944700 1 Processor -- start 2024-01-21T20:05:21.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:21.058+0000 7f4b4f944700 1 -- start start 2024-01-21T20:05:21.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:21.058+0000 7f4b4f944700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b480f5410 0x7f4b481093c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:21.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:21.058+0000 7f4b4f944700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b48060280 con 0x7f4b480f5410 2024-01-21T20:05:21.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:21.059+0000 7f4b4d6e0700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b480f5410 0x7f4b481093c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:21.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:21.059+0000 7f4b4d6e0700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b480f5410 0x7f4b481093c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:40228/0 (socket says 172.21.15.146:40228) 2024-01-21T20:05:21.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:21.059+0000 7f4b4d6e0700 1 -- 172.21.15.146:0/2076854516 learned_addr learned my addr 172.21.15.146:0/2076854516 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:05:21.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:21.059+0000 7f4b4d6e0700 1 -- 172.21.15.146:0/2076854516 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b44005040 con 0x7f4b480f5410 2024-01-21T20:05:21.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:21.059+0000 7f4b4d6e0700 1 --2- 172.21.15.146:0/2076854516 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b480f5410 0x7f4b481093c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f4b44013510 tx=0x7f4b4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:21.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:21.059+0000 7f4b3dffb700 1 -- 172.21.15.146:0/2076854516 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b44009320 con 0x7f4b480f5410 2024-01-21T20:05:21.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:21.059+0000 7f4b4f944700 1 -- 172.21.15.146:0/2076854516 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b48060480 con 0x7f4b480f5410 2024-01-21T20:05:21.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:21.059+0000 7f4b3dffb700 1 -- 172.21.15.146:0/2076854516 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4b44013f00 con 0x7f4b480f5410 2024-01-21T20:05:21.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:21.059+0000 7f4b3dffb700 1 -- 172.21.15.146:0/2076854516 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b44014d90 con 0x7f4b480f5410 2024-01-21T20:05:21.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:21.059+0000 7f4b4f944700 1 -- 172.21.15.146:0/2076854516 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b48109af0 con 0x7f4b480f5410 2024-01-21T20:05:21.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:21.060+0000 7f4b3dffb700 1 -- 172.21.15.146:0/2076854516 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4b440095f0 con 0x7f4b480f5410 2024-01-21T20:05:21.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:21.060+0000 7f4b4f944700 1 -- 172.21.15.146:0/2076854516 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b48045ad0 con 0x7f4b480f5410 2024-01-21T20:05:21.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:21.060+0000 7f4b3dffb700 1 --2- 172.21.15.146:0/2076854516 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f4b3402ffb0 0x7f4b34032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:21.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:21.060+0000 7f4b3dffb700 1 -- 172.21.15.146:0/2076854516 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4b44040fe0 con 0x7f4b480f5410 2024-01-21T20:05:21.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:21.061+0000 7f4b4cedf700 1 --2- 172.21.15.146:0/2076854516 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f4b3402ffb0 0x7f4b34032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:21.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:21.061+0000 7f4b4cedf700 1 --2- 172.21.15.146:0/2076854516 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f4b3402ffb0 0x7f4b34032470 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f4b380096a0 tx=0x7f4b38006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:21.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:21.065+0000 7f4b3dffb700 1 -- 172.21.15.146:0/2076854516 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4b4406d2b0 con 0x7f4b480f5410 2024-01-21T20:05:21.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:20 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:05:19.183623+0000 mgr.smithi053.fgsmol (mgr.14164) 191 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:21.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:21.269+0000 7f4b4f944700 1 -- 172.21.15.146:0/2076854516 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4b4810a340 con 0x7f4b480f5410 2024-01-21T20:05:21.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:21.270+0000 7f4b3dffb700 1 -- 172.21.15.146:0/2076854516 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4b44014420 con 0x7f4b480f5410 2024-01-21T20:05:21.261 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:05:21.261 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:05:21.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:21.271+0000 7f4b4f944700 1 -- 172.21.15.146:0/2076854516 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f4b3402ffb0 msgr2=0x7f4b34032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:21.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:21.271+0000 7f4b4f944700 1 --2- 172.21.15.146:0/2076854516 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f4b3402ffb0 0x7f4b34032470 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f4b380096a0 tx=0x7f4b38006b40).stop 2024-01-21T20:05:21.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:21.271+0000 7f4b4f944700 1 -- 172.21.15.146:0/2076854516 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b480f5410 msgr2=0x7f4b481093c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:21.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:21.271+0000 7f4b4f944700 1 --2- 172.21.15.146:0/2076854516 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b480f5410 0x7f4b481093c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f4b44013510 tx=0x7f4b4400b040).stop 2024-01-21T20:05:21.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:21.272+0000 7f4b4f944700 1 -- 172.21.15.146:0/2076854516 shutdown_connections 2024-01-21T20:05:21.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:21.272+0000 7f4b4f944700 1 --2- 172.21.15.146:0/2076854516 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f4b3402ffb0 0x7f4b34032470 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:21.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:21.272+0000 7f4b4f944700 1 --2- 172.21.15.146:0/2076854516 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b480f5410 0x7f4b481093c0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:21.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:21.272+0000 7f4b4f944700 1 -- 172.21.15.146:0/2076854516 >> 172.21.15.146:0/2076854516 conn(0x7f4b480f0b70 msgr2=0x7f4b480f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:21.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:21.272+0000 7f4b4f944700 1 -- 172.21.15.146:0/2076854516 shutdown_connections 2024-01-21T20:05:21.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:21.272+0000 7f4b4f944700 1 -- 172.21.15.146:0/2076854516 wait complete. 2024-01-21T20:05:21.263 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:05:22.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:21 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:05:21.271260+0000 mon.smithi053 (mon.0) 258 : audit [DBG] from='client.? 172.21.15.146:0/2076854516' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:05:22.712 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:05:22.712 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:05:23.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:05:21.183937+0000 mgr.smithi053.fgsmol (mgr.14164) 192 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:23.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:23.912+0000 7f8031918700 1 -- 172.21.15.146:0/3006544019 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f802c0f45b0 msgr2=0x7f802c0f49d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:23.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:23.912+0000 7f8031918700 1 --2- 172.21.15.146:0/3006544019 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f802c0f45b0 0x7f802c0f49d0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f801c009fb0 tx=0x7f801c009b30).stop 2024-01-21T20:05:23.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:23.912+0000 7f8031918700 1 -- 172.21.15.146:0/3006544019 shutdown_connections 2024-01-21T20:05:23.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:23.912+0000 7f8031918700 1 --2- 172.21.15.146:0/3006544019 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f802c0f45b0 0x7f802c0f49d0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:23.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:23.912+0000 7f8031918700 1 -- 172.21.15.146:0/3006544019 >> 172.21.15.146:0/3006544019 conn(0x7f802c0f00f0 msgr2=0x7f802c0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:23.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:23.912+0000 7f8031918700 1 -- 172.21.15.146:0/3006544019 shutdown_connections 2024-01-21T20:05:23.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:23.912+0000 7f8031918700 1 -- 172.21.15.146:0/3006544019 wait complete. 2024-01-21T20:05:23.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:23.913+0000 7f8031918700 1 Processor -- start 2024-01-21T20:05:23.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:23.913+0000 7f8031918700 1 -- start start 2024-01-21T20:05:23.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:23.913+0000 7f8031918700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f802c0f45b0 0x7f802c106550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:23.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:23.913+0000 7f8031918700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f802c106a40 con 0x7f802c0f45b0 2024-01-21T20:05:23.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:23.913+0000 7f802affd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f802c0f45b0 0x7f802c106550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:23.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:23.913+0000 7f802affd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f802c0f45b0 0x7f802c106550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:40234/0 (socket says 172.21.15.146:40234) 2024-01-21T20:05:23.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:23.913+0000 7f802affd700 1 -- 172.21.15.146:0/1366686885 learned_addr learned my addr 172.21.15.146:0/1366686885 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:05:23.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:23.914+0000 7f802affd700 1 -- 172.21.15.146:0/1366686885 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f801c005040 con 0x7f802c0f45b0 2024-01-21T20:05:23.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:23.914+0000 7f802affd700 1 --2- 172.21.15.146:0/1366686885 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f802c0f45b0 0x7f802c106550 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f801c013510 tx=0x7f801c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:23.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:23.914+0000 7f80137fe700 1 -- 172.21.15.146:0/1366686885 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f801c009320 con 0x7f802c0f45b0 2024-01-21T20:05:23.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:23.914+0000 7f8031918700 1 -- 172.21.15.146:0/1366686885 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f802c106c40 con 0x7f802c0f45b0 2024-01-21T20:05:23.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:23.914+0000 7f80137fe700 1 -- 172.21.15.146:0/1366686885 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f801c013f00 con 0x7f802c0f45b0 2024-01-21T20:05:23.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:23.914+0000 7f80137fe700 1 -- 172.21.15.146:0/1366686885 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f801c014d90 con 0x7f802c0f45b0 2024-01-21T20:05:23.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:23.914+0000 7f8031918700 1 -- 172.21.15.146:0/1366686885 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f802c1070d0 con 0x7f802c0f45b0 2024-01-21T20:05:23.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:23.915+0000 7f80137fe700 1 -- 172.21.15.146:0/1366686885 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f801c0095f0 con 0x7f802c0f45b0 2024-01-21T20:05:23.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:23.915+0000 7f8031918700 1 -- 172.21.15.146:0/1366686885 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f802c045ad0 con 0x7f802c0f45b0 2024-01-21T20:05:23.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:23.915+0000 7f80137fe700 1 --2- 172.21.15.146:0/1366686885 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f801402ff60 0x7f8014032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:23.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:23.915+0000 7f80137fe700 1 -- 172.21.15.146:0/1366686885 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f801c0410a0 con 0x7f802c0f45b0 2024-01-21T20:05:23.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:23.915+0000 7f802a7fc700 1 --2- 172.21.15.146:0/1366686885 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f801402ff60 0x7f8014032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:23.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:23.916+0000 7f802a7fc700 1 --2- 172.21.15.146:0/1366686885 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f801402ff60 0x7f8014032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f80200096a0 tx=0x7f8020006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:23.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:23.920+0000 7f80137fe700 1 -- 172.21.15.146:0/1366686885 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f801c01d7f0 con 0x7f802c0f45b0 2024-01-21T20:05:24.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:24.123+0000 7f8031918700 1 -- 172.21.15.146:0/1366686885 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f802c107ed0 con 0x7f802c0f45b0 2024-01-21T20:05:24.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:24.124+0000 7f80137fe700 1 -- 172.21.15.146:0/1366686885 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f801c010230 con 0x7f802c0f45b0 2024-01-21T20:05:24.115 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:05:24.115 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:05:24.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:24.126+0000 7f8031918700 1 -- 172.21.15.146:0/1366686885 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f801402ff60 msgr2=0x7f8014032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:24.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:24.126+0000 7f8031918700 1 --2- 172.21.15.146:0/1366686885 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f801402ff60 0x7f8014032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f80200096a0 tx=0x7f8020006b40).stop 2024-01-21T20:05:24.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:24.126+0000 7f8031918700 1 -- 172.21.15.146:0/1366686885 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f802c0f45b0 msgr2=0x7f802c106550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:24.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:24.126+0000 7f8031918700 1 --2- 172.21.15.146:0/1366686885 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f802c0f45b0 0x7f802c106550 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f801c013510 tx=0x7f801c00b040).stop 2024-01-21T20:05:24.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:24.126+0000 7f8031918700 1 -- 172.21.15.146:0/1366686885 shutdown_connections 2024-01-21T20:05:24.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:24.126+0000 7f8031918700 1 --2- 172.21.15.146:0/1366686885 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f801402ff60 0x7f8014032420 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:24.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:24.126+0000 7f8031918700 1 --2- 172.21.15.146:0/1366686885 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f802c0f45b0 0x7f802c106550 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:24.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:24.126+0000 7f8031918700 1 -- 172.21.15.146:0/1366686885 >> 172.21.15.146:0/1366686885 conn(0x7f802c0f00f0 msgr2=0x7f802c0f0c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:24.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:24.126+0000 7f8031918700 1 -- 172.21.15.146:0/1366686885 shutdown_connections 2024-01-21T20:05:24.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:24.126+0000 7f8031918700 1 -- 172.21.15.146:0/1366686885 wait complete. 2024-01-21T20:05:24.118 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:05:25.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:24 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:05:23.184308+0000 mgr.smithi053.fgsmol (mgr.14164) 193 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:25.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:24 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:05:24.125472+0000 mon.smithi053 (mon.0) 259 : audit [DBG] from='client.? 172.21.15.146:0/1366686885' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:05:25.535 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:05:25.535 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:05:26.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:26.788+0000 7f295dc25700 1 -- 172.21.15.146:0/2122315109 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29580f6510 msgr2=0x7f29580f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:26.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:26.788+0000 7f295dc25700 1 --2- 172.21.15.146:0/2122315109 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29580f6510 0x7f29580f6930 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f2940009fb0 tx=0x7f2940009b30).stop 2024-01-21T20:05:26.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:26.788+0000 7f295dc25700 1 -- 172.21.15.146:0/2122315109 shutdown_connections 2024-01-21T20:05:26.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:26.788+0000 7f295dc25700 1 --2- 172.21.15.146:0/2122315109 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29580f6510 0x7f29580f6930 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:26.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:26.788+0000 7f295dc25700 1 -- 172.21.15.146:0/2122315109 >> 172.21.15.146:0/2122315109 conn(0x7f29580f1c70 msgr2=0x7f29580f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:26.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:26.788+0000 7f295dc25700 1 -- 172.21.15.146:0/2122315109 shutdown_connections 2024-01-21T20:05:26.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:26.788+0000 7f295dc25700 1 -- 172.21.15.146:0/2122315109 wait complete. 2024-01-21T20:05:26.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:26.789+0000 7f295dc25700 1 Processor -- start 2024-01-21T20:05:26.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:26.789+0000 7f295dc25700 1 -- start start 2024-01-21T20:05:26.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:26.789+0000 7f295dc25700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29580f6510 0x7f2958106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:26.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:26.789+0000 7f295dc25700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2958106c20 con 0x7f29580f6510 2024-01-21T20:05:26.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:26.789+0000 7f29577fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29580f6510 0x7f2958106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:26.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:26.789+0000 7f29577fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29580f6510 0x7f2958106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:40254/0 (socket says 172.21.15.146:40254) 2024-01-21T20:05:26.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:26.789+0000 7f29577fe700 1 -- 172.21.15.146:0/2484332690 learned_addr learned my addr 172.21.15.146:0/2484332690 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:05:26.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:26.790+0000 7f29577fe700 1 -- 172.21.15.146:0/2484332690 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2940005040 con 0x7f29580f6510 2024-01-21T20:05:26.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:26.790+0000 7f29577fe700 1 --2- 172.21.15.146:0/2484332690 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29580f6510 0x7f2958106730 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f2940013510 tx=0x7f294000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:26.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:26.790+0000 7f294ffff700 1 -- 172.21.15.146:0/2484332690 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2940009320 con 0x7f29580f6510 2024-01-21T20:05:26.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:26.790+0000 7f295dc25700 1 -- 172.21.15.146:0/2484332690 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2958106e20 con 0x7f29580f6510 2024-01-21T20:05:26.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:26.790+0000 7f294ffff700 1 -- 172.21.15.146:0/2484332690 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2940013f00 con 0x7f29580f6510 2024-01-21T20:05:26.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:26.790+0000 7f294ffff700 1 -- 172.21.15.146:0/2484332690 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2940014d90 con 0x7f29580f6510 2024-01-21T20:05:26.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:26.790+0000 7f295dc25700 1 -- 172.21.15.146:0/2484332690 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29581072b0 con 0x7f29580f6510 2024-01-21T20:05:26.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:26.791+0000 7f294ffff700 1 -- 172.21.15.146:0/2484332690 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f29400263a0 con 0x7f29580f6510 2024-01-21T20:05:26.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:26.791+0000 7f294ffff700 1 --2- 172.21.15.146:0/2484332690 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f294402ff60 0x7f2944032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:26.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:26.791+0000 7f294ffff700 1 -- 172.21.15.146:0/2484332690 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2940040ff0 con 0x7f29580f6510 2024-01-21T20:05:26.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:26.791+0000 7f295dc25700 1 -- 172.21.15.146:0/2484332690 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2958059100 con 0x7f29580f6510 2024-01-21T20:05:26.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:26.791+0000 7f2956ffd700 1 --2- 172.21.15.146:0/2484332690 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f294402ff60 0x7f2944032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:26.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:26.792+0000 7f2956ffd700 1 --2- 172.21.15.146:0/2484332690 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f294402ff60 0x7f2944032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f29480096a0 tx=0x7f2948006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:26.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:26.796+0000 7f294ffff700 1 -- 172.21.15.146:0/2484332690 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2940012920 con 0x7f29580f6510 2024-01-21T20:05:26.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:27.000+0000 7f295dc25700 1 -- 172.21.15.146:0/2484332690 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f29580fbf60 con 0x7f29580f6510 2024-01-21T20:05:26.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:27.002+0000 7f294ffff700 1 -- 172.21.15.146:0/2484332690 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f294002b020 con 0x7f29580f6510 2024-01-21T20:05:26.992 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:05:26.992 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:05:26.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:27.002+0000 7f295dc25700 1 -- 172.21.15.146:0/2484332690 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f294402ff60 msgr2=0x7f2944032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:26.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:27.002+0000 7f295dc25700 1 --2- 172.21.15.146:0/2484332690 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f294402ff60 0x7f2944032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f29480096a0 tx=0x7f2948006b40).stop 2024-01-21T20:05:26.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:27.002+0000 7f295dc25700 1 -- 172.21.15.146:0/2484332690 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29580f6510 msgr2=0x7f2958106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:26.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:27.002+0000 7f295dc25700 1 --2- 172.21.15.146:0/2484332690 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29580f6510 0x7f2958106730 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f2940013510 tx=0x7f294000b040).stop 2024-01-21T20:05:26.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:27.002+0000 7f295dc25700 1 -- 172.21.15.146:0/2484332690 shutdown_connections 2024-01-21T20:05:26.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:27.002+0000 7f295dc25700 1 --2- 172.21.15.146:0/2484332690 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f294402ff60 0x7f2944032420 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:26.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:27.002+0000 7f295dc25700 1 --2- 172.21.15.146:0/2484332690 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29580f6510 0x7f2958106730 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:26.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:27.002+0000 7f295dc25700 1 -- 172.21.15.146:0/2484332690 >> 172.21.15.146:0/2484332690 conn(0x7f29580f1c70 msgr2=0x7f29580f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:26.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:27.002+0000 7f295dc25700 1 -- 172.21.15.146:0/2484332690 shutdown_connections 2024-01-21T20:05:26.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:27.003+0000 7f295dc25700 1 -- 172.21.15.146:0/2484332690 wait complete. 2024-01-21T20:05:26.994 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:05:27.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:26 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:05:25.184628+0000 mgr.smithi053.fgsmol (mgr.14164) 194 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:28.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:27 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:05:27.002120+0000 mon.smithi053 (mon.0) 260 : audit [DBG] from='client.? 172.21.15.146:0/2484332690' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:05:28.418 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:05:28.418 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:05:29.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:28 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:05:27.184988+0000 mgr.smithi053.fgsmol (mgr.14164) 195 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:29.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:29.622+0000 7f4d6bba5700 1 -- 172.21.15.146:0/3211787326 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d640f5410 msgr2=0x7f4d640f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:29.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:29.622+0000 7f4d6bba5700 1 --2- 172.21.15.146:0/3211787326 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d640f5410 0x7f4d640f5830 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f4d60009fb0 tx=0x7f4d60009b30).stop 2024-01-21T20:05:29.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:29.622+0000 7f4d6bba5700 1 -- 172.21.15.146:0/3211787326 shutdown_connections 2024-01-21T20:05:29.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:29.622+0000 7f4d6bba5700 1 --2- 172.21.15.146:0/3211787326 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d640f5410 0x7f4d640f5830 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:29.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:29.622+0000 7f4d6bba5700 1 -- 172.21.15.146:0/3211787326 >> 172.21.15.146:0/3211787326 conn(0x7f4d640f0b70 msgr2=0x7f4d640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:29.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:29.622+0000 7f4d6bba5700 1 -- 172.21.15.146:0/3211787326 shutdown_connections 2024-01-21T20:05:29.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:29.622+0000 7f4d6bba5700 1 -- 172.21.15.146:0/3211787326 wait complete. 2024-01-21T20:05:29.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:29.623+0000 7f4d6bba5700 1 Processor -- start 2024-01-21T20:05:29.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:29.623+0000 7f4d6bba5700 1 -- start start 2024-01-21T20:05:29.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:29.623+0000 7f4d6bba5700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d640f5410 0x7f4d64106750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:29.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:29.623+0000 7f4d6bba5700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d64106c40 con 0x7f4d640f5410 2024-01-21T20:05:29.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:29.623+0000 7f4d69941700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d640f5410 0x7f4d64106750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:29.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:29.623+0000 7f4d69941700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d640f5410 0x7f4d64106750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:59432/0 (socket says 172.21.15.146:59432) 2024-01-21T20:05:29.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:29.623+0000 7f4d69941700 1 -- 172.21.15.146:0/3944776235 learned_addr learned my addr 172.21.15.146:0/3944776235 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:05:29.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:29.624+0000 7f4d69941700 1 -- 172.21.15.146:0/3944776235 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d60005040 con 0x7f4d640f5410 2024-01-21T20:05:29.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:29.624+0000 7f4d69941700 1 --2- 172.21.15.146:0/3944776235 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d640f5410 0x7f4d64106750 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f4d60009f80 tx=0x7f4d6000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:29.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:29.624+0000 7f4d5a7fc700 1 -- 172.21.15.146:0/3944776235 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d60009320 con 0x7f4d640f5410 2024-01-21T20:05:29.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:29.624+0000 7f4d6bba5700 1 -- 172.21.15.146:0/3944776235 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d64106e40 con 0x7f4d640f5410 2024-01-21T20:05:29.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:29.624+0000 7f4d5a7fc700 1 -- 172.21.15.146:0/3944776235 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4d60013ec0 con 0x7f4d640f5410 2024-01-21T20:05:29.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:29.624+0000 7f4d5a7fc700 1 -- 172.21.15.146:0/3944776235 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d6001ece0 con 0x7f4d640f5410 2024-01-21T20:05:29.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:29.624+0000 7f4d6bba5700 1 -- 172.21.15.146:0/3944776235 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d641072d0 con 0x7f4d640f5410 2024-01-21T20:05:29.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:29.625+0000 7f4d5a7fc700 1 -- 172.21.15.146:0/3944776235 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4d6001e420 con 0x7f4d640f5410 2024-01-21T20:05:29.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:29.625+0000 7f4d6bba5700 1 -- 172.21.15.146:0/3944776235 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d64045ad0 con 0x7f4d640f5410 2024-01-21T20:05:29.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:29.625+0000 7f4d5a7fc700 1 --2- 172.21.15.146:0/3944776235 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f4d5002ff60 0x7f4d50032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:29.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:29.625+0000 7f4d5a7fc700 1 -- 172.21.15.146:0/3944776235 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4d60040e40 con 0x7f4d640f5410 2024-01-21T20:05:29.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:29.625+0000 7f4d69140700 1 --2- 172.21.15.146:0/3944776235 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f4d5002ff60 0x7f4d50032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:29.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:29.626+0000 7f4d69140700 1 --2- 172.21.15.146:0/3944776235 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f4d5002ff60 0x7f4d50032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f4d540096a0 tx=0x7f4d54006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:29.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:29.630+0000 7f4d5a7fc700 1 -- 172.21.15.146:0/3944776235 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4d6001e730 con 0x7f4d640f5410 2024-01-21T20:05:29.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:29.835+0000 7f4d6bba5700 1 -- 172.21.15.146:0/3944776235 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4d64108180 con 0x7f4d640f5410 2024-01-21T20:05:29.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:29.835+0000 7f4d5a7fc700 1 -- 172.21.15.146:0/3944776235 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4d60042030 con 0x7f4d640f5410 2024-01-21T20:05:29.826 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:05:29.827 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:05:29.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:29.837+0000 7f4d6bba5700 1 -- 172.21.15.146:0/3944776235 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f4d5002ff60 msgr2=0x7f4d50032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:29.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:29.837+0000 7f4d6bba5700 1 --2- 172.21.15.146:0/3944776235 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f4d5002ff60 0x7f4d50032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f4d540096a0 tx=0x7f4d54006b40).stop 2024-01-21T20:05:29.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:29.837+0000 7f4d6bba5700 1 -- 172.21.15.146:0/3944776235 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d640f5410 msgr2=0x7f4d64106750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:29.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:29.837+0000 7f4d6bba5700 1 --2- 172.21.15.146:0/3944776235 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d640f5410 0x7f4d64106750 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f4d60009f80 tx=0x7f4d6000b040).stop 2024-01-21T20:05:29.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:29.837+0000 7f4d6bba5700 1 -- 172.21.15.146:0/3944776235 shutdown_connections 2024-01-21T20:05:29.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:29.837+0000 7f4d6bba5700 1 --2- 172.21.15.146:0/3944776235 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f4d5002ff60 0x7f4d50032420 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:29.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:29.837+0000 7f4d6bba5700 1 --2- 172.21.15.146:0/3944776235 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d640f5410 0x7f4d64106750 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:29.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:29.837+0000 7f4d6bba5700 1 -- 172.21.15.146:0/3944776235 >> 172.21.15.146:0/3944776235 conn(0x7f4d640f0b70 msgr2=0x7f4d6405fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:29.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:29.837+0000 7f4d6bba5700 1 -- 172.21.15.146:0/3944776235 shutdown_connections 2024-01-21T20:05:29.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:29.837+0000 7f4d6bba5700 1 -- 172.21.15.146:0/3944776235 wait complete. 2024-01-21T20:05:29.829 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:05:30.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:29 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:05:29.836736+0000 mon.smithi053 (mon.0) 261 : audit [DBG] from='client.? 172.21.15.146:0/3944776235' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:05:31.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:30 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:05:29.185403+0000 mgr.smithi053.fgsmol (mgr.14164) 196 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:31.251 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:05:31.251 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:05:32.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:32.469+0000 7fa65ceca700 1 -- 172.21.15.146:0/1123823345 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6580f5410 msgr2=0x7fa6580f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:32.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:32.469+0000 7fa65ceca700 1 --2- 172.21.15.146:0/1123823345 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6580f5410 0x7fa6580f5830 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7fa648009fb0 tx=0x7fa648009b30).stop 2024-01-21T20:05:32.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:32.469+0000 7fa65ceca700 1 -- 172.21.15.146:0/1123823345 shutdown_connections 2024-01-21T20:05:32.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:32.469+0000 7fa65ceca700 1 --2- 172.21.15.146:0/1123823345 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6580f5410 0x7fa6580f5830 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:32.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:32.469+0000 7fa65ceca700 1 -- 172.21.15.146:0/1123823345 >> 172.21.15.146:0/1123823345 conn(0x7fa6580f0b70 msgr2=0x7fa6580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:32.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:32.469+0000 7fa65ceca700 1 -- 172.21.15.146:0/1123823345 shutdown_connections 2024-01-21T20:05:32.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:32.469+0000 7fa65ceca700 1 -- 172.21.15.146:0/1123823345 wait complete. 2024-01-21T20:05:32.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:32.470+0000 7fa65ceca700 1 Processor -- start 2024-01-21T20:05:32.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:32.470+0000 7fa65ceca700 1 -- start start 2024-01-21T20:05:32.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:32.470+0000 7fa65ceca700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6580f5410 0x7fa658106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:32.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:32.470+0000 7fa65ceca700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa658106c10 con 0x7fa6580f5410 2024-01-21T20:05:32.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:32.470+0000 7fa65659c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6580f5410 0x7fa658106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:32.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:32.470+0000 7fa65659c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6580f5410 0x7fa658106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:59454/0 (socket says 172.21.15.146:59454) 2024-01-21T20:05:32.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:32.470+0000 7fa65659c700 1 -- 172.21.15.146:0/906686962 learned_addr learned my addr 172.21.15.146:0/906686962 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:05:32.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:32.471+0000 7fa65659c700 1 -- 172.21.15.146:0/906686962 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa648005040 con 0x7fa6580f5410 2024-01-21T20:05:32.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:32.471+0000 7fa65659c700 1 --2- 172.21.15.146:0/906686962 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6580f5410 0x7fa658106720 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fa648013510 tx=0x7fa64800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:32.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:32.471+0000 7fa64effd700 1 -- 172.21.15.146:0/906686962 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa648009320 con 0x7fa6580f5410 2024-01-21T20:05:32.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:32.471+0000 7fa65ceca700 1 -- 172.21.15.146:0/906686962 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa658106e10 con 0x7fa6580f5410 2024-01-21T20:05:32.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:32.471+0000 7fa64effd700 1 -- 172.21.15.146:0/906686962 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa648013f00 con 0x7fa6580f5410 2024-01-21T20:05:32.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:32.471+0000 7fa64effd700 1 -- 172.21.15.146:0/906686962 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa648014d90 con 0x7fa6580f5410 2024-01-21T20:05:32.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:32.471+0000 7fa65ceca700 1 -- 172.21.15.146:0/906686962 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6581072a0 con 0x7fa6580f5410 2024-01-21T20:05:32.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:32.472+0000 7fa65ceca700 1 -- 172.21.15.146:0/906686962 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa658045ad0 con 0x7fa6580f5410 2024-01-21T20:05:32.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:32.472+0000 7fa64effd700 1 -- 172.21.15.146:0/906686962 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa6480095f0 con 0x7fa6580f5410 2024-01-21T20:05:32.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:32.472+0000 7fa64effd700 1 --2- 172.21.15.146:0/906686962 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa64402ff60 0x7fa644032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:32.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:32.472+0000 7fa64effd700 1 -- 172.21.15.146:0/906686962 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa6480410c0 con 0x7fa6580f5410 2024-01-21T20:05:32.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:32.472+0000 7fa655d9b700 1 --2- 172.21.15.146:0/906686962 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa64402ff60 0x7fa644032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:32.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:32.473+0000 7fa655d9b700 1 --2- 172.21.15.146:0/906686962 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa64402ff60 0x7fa644032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fa6400096a0 tx=0x7fa640006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:32.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:32.477+0000 7fa64effd700 1 -- 172.21.15.146:0/906686962 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa648014420 con 0x7fa6580f5410 2024-01-21T20:05:32.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:32.680+0000 7fa65ceca700 1 -- 172.21.15.146:0/906686962 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa658108150 con 0x7fa6580f5410 2024-01-21T20:05:32.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:32.680+0000 7fa64effd700 1 -- 172.21.15.146:0/906686962 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa64801d7f0 con 0x7fa6580f5410 2024-01-21T20:05:32.671 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:05:32.671 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:05:32.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:32.682+0000 7fa65ceca700 1 -- 172.21.15.146:0/906686962 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa64402ff60 msgr2=0x7fa644032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:32.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:32.682+0000 7fa65ceca700 1 --2- 172.21.15.146:0/906686962 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa64402ff60 0x7fa644032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fa6400096a0 tx=0x7fa640006b40).stop 2024-01-21T20:05:32.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:32.682+0000 7fa65ceca700 1 -- 172.21.15.146:0/906686962 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6580f5410 msgr2=0x7fa658106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:32.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:32.682+0000 7fa65ceca700 1 --2- 172.21.15.146:0/906686962 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6580f5410 0x7fa658106720 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fa648013510 tx=0x7fa64800b040).stop 2024-01-21T20:05:32.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:32.682+0000 7fa65ceca700 1 -- 172.21.15.146:0/906686962 shutdown_connections 2024-01-21T20:05:32.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:32.682+0000 7fa65ceca700 1 --2- 172.21.15.146:0/906686962 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa64402ff60 0x7fa644032420 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:32.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:32.682+0000 7fa65ceca700 1 --2- 172.21.15.146:0/906686962 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6580f5410 0x7fa658106720 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:32.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:32.682+0000 7fa65ceca700 1 -- 172.21.15.146:0/906686962 >> 172.21.15.146:0/906686962 conn(0x7fa6580f0b70 msgr2=0x7fa65805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:32.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:32.682+0000 7fa65ceca700 1 -- 172.21.15.146:0/906686962 shutdown_connections 2024-01-21T20:05:32.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:32.682+0000 7fa65ceca700 1 -- 172.21.15.146:0/906686962 wait complete. 2024-01-21T20:05:32.674 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:05:33.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:32 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:05:31.185779+0000 mgr.smithi053.fgsmol (mgr.14164) 197 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:33.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:32 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:05:32.681849+0000 mon.smithi053 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.146:0/906686962' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:05:34.092 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:05:34.093 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:05:35.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:34 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:05:33.186104+0000 mgr.smithi053.fgsmol (mgr.14164) 198 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:35.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:35.276+0000 7ff927854700 1 -- 172.21.15.146:0/1826040769 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9200f6510 msgr2=0x7ff9200f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:35.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:35.276+0000 7ff927854700 1 --2- 172.21.15.146:0/1826040769 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9200f6510 0x7ff9200f6930 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7ff914009fb0 tx=0x7ff914009b30).stop 2024-01-21T20:05:35.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:35.276+0000 7ff927854700 1 -- 172.21.15.146:0/1826040769 shutdown_connections 2024-01-21T20:05:35.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:35.276+0000 7ff927854700 1 --2- 172.21.15.146:0/1826040769 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9200f6510 0x7ff9200f6930 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:35.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:35.276+0000 7ff927854700 1 -- 172.21.15.146:0/1826040769 >> 172.21.15.146:0/1826040769 conn(0x7ff9200f1c70 msgr2=0x7ff9200f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:35.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:35.276+0000 7ff927854700 1 -- 172.21.15.146:0/1826040769 shutdown_connections 2024-01-21T20:05:35.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:35.276+0000 7ff927854700 1 -- 172.21.15.146:0/1826040769 wait complete. 2024-01-21T20:05:35.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:35.277+0000 7ff927854700 1 Processor -- start 2024-01-21T20:05:35.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:35.277+0000 7ff927854700 1 -- start start 2024-01-21T20:05:35.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:35.277+0000 7ff927854700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9200f6510 0x7ff9201066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:35.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:35.277+0000 7ff927854700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff920106bc0 con 0x7ff9200f6510 2024-01-21T20:05:35.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:35.278+0000 7ff9255f0700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9200f6510 0x7ff9201066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:35.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:35.278+0000 7ff9255f0700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9200f6510 0x7ff9201066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:59472/0 (socket says 172.21.15.146:59472) 2024-01-21T20:05:35.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:35.278+0000 7ff9255f0700 1 -- 172.21.15.146:0/97620369 learned_addr learned my addr 172.21.15.146:0/97620369 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:05:35.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:35.278+0000 7ff9255f0700 1 -- 172.21.15.146:0/97620369 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff914005040 con 0x7ff9200f6510 2024-01-21T20:05:35.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:35.278+0000 7ff9255f0700 1 --2- 172.21.15.146:0/97620369 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9200f6510 0x7ff9201066d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7ff914013510 tx=0x7ff91400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:35.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:35.278+0000 7ff911ffb700 1 -- 172.21.15.146:0/97620369 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff914009320 con 0x7ff9200f6510 2024-01-21T20:05:35.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:35.278+0000 7ff927854700 1 -- 172.21.15.146:0/97620369 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff920106dc0 con 0x7ff9200f6510 2024-01-21T20:05:35.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:35.278+0000 7ff911ffb700 1 -- 172.21.15.146:0/97620369 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff914013f00 con 0x7ff9200f6510 2024-01-21T20:05:35.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:35.278+0000 7ff911ffb700 1 -- 172.21.15.146:0/97620369 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff914014d90 con 0x7ff9200f6510 2024-01-21T20:05:35.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:35.278+0000 7ff927854700 1 -- 172.21.15.146:0/97620369 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff920107250 con 0x7ff9200f6510 2024-01-21T20:05:35.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:35.279+0000 7ff911ffb700 1 -- 172.21.15.146:0/97620369 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff9140095f0 con 0x7ff9200f6510 2024-01-21T20:05:35.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:35.279+0000 7ff927854700 1 -- 172.21.15.146:0/97620369 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff920045ad0 con 0x7ff9200f6510 2024-01-21T20:05:35.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:35.279+0000 7ff911ffb700 1 --2- 172.21.15.146:0/97620369 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff90c030000 0x7ff90c0324c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:35.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:35.279+0000 7ff911ffb700 1 -- 172.21.15.146:0/97620369 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff914041040 con 0x7ff9200f6510 2024-01-21T20:05:35.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:35.280+0000 7ff924def700 1 --2- 172.21.15.146:0/97620369 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff90c030000 0x7ff90c0324c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:35.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:35.280+0000 7ff924def700 1 --2- 172.21.15.146:0/97620369 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff90c030000 0x7ff90c0324c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7ff91c0096a0 tx=0x7ff91c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:35.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:35.285+0000 7ff911ffb700 1 -- 172.21.15.146:0/97620369 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff914014420 con 0x7ff9200f6510 2024-01-21T20:05:35.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:35.490+0000 7ff927854700 1 -- 172.21.15.146:0/97620369 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff920108100 con 0x7ff9200f6510 2024-01-21T20:05:35.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:35.491+0000 7ff911ffb700 1 -- 172.21.15.146:0/97620369 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff91401d7f0 con 0x7ff9200f6510 2024-01-21T20:05:35.482 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:05:35.482 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:05:35.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:35.492+0000 7ff927854700 1 -- 172.21.15.146:0/97620369 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff90c030000 msgr2=0x7ff90c0324c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:35.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:35.492+0000 7ff927854700 1 --2- 172.21.15.146:0/97620369 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff90c030000 0x7ff90c0324c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7ff91c0096a0 tx=0x7ff91c006b40).stop 2024-01-21T20:05:35.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:35.492+0000 7ff927854700 1 -- 172.21.15.146:0/97620369 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9200f6510 msgr2=0x7ff9201066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:35.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:35.492+0000 7ff927854700 1 --2- 172.21.15.146:0/97620369 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9200f6510 0x7ff9201066d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7ff914013510 tx=0x7ff91400b040).stop 2024-01-21T20:05:35.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:35.492+0000 7ff927854700 1 -- 172.21.15.146:0/97620369 shutdown_connections 2024-01-21T20:05:35.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:35.492+0000 7ff927854700 1 --2- 172.21.15.146:0/97620369 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff90c030000 0x7ff90c0324c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:35.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:35.493+0000 7ff927854700 1 --2- 172.21.15.146:0/97620369 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9200f6510 0x7ff9201066d0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:35.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:35.493+0000 7ff927854700 1 -- 172.21.15.146:0/97620369 >> 172.21.15.146:0/97620369 conn(0x7ff9200f1c70 msgr2=0x7ff9200f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:35.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:35.493+0000 7ff927854700 1 -- 172.21.15.146:0/97620369 shutdown_connections 2024-01-21T20:05:35.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:35.493+0000 7ff927854700 1 -- 172.21.15.146:0/97620369 wait complete. 2024-01-21T20:05:35.485 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:05:36.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:35 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:05:35.492298+0000 mon.smithi053 (mon.0) 263 : audit [DBG] from='client.? 172.21.15.146:0/97620369' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:05:36.923 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:05:36.923 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:05:37.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:36 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:05:35.186443+0000 mgr.smithi053.fgsmol (mgr.14164) 199 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:38.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:38.190+0000 7f6196bb3700 1 -- 172.21.15.146:0/680856765 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f61900f5410 msgr2=0x7f61900f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:38.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:38.190+0000 7f6196bb3700 1 --2- 172.21.15.146:0/680856765 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f61900f5410 0x7f61900f5830 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f6184009fb0 tx=0x7f6184009b30).stop 2024-01-21T20:05:38.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:38.190+0000 7f6196bb3700 1 -- 172.21.15.146:0/680856765 shutdown_connections 2024-01-21T20:05:38.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:38.190+0000 7f6196bb3700 1 --2- 172.21.15.146:0/680856765 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f61900f5410 0x7f61900f5830 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:38.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:38.190+0000 7f6196bb3700 1 -- 172.21.15.146:0/680856765 >> 172.21.15.146:0/680856765 conn(0x7f61900f0b70 msgr2=0x7f61900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:38.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:38.190+0000 7f6196bb3700 1 -- 172.21.15.146:0/680856765 shutdown_connections 2024-01-21T20:05:38.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:38.190+0000 7f6196bb3700 1 -- 172.21.15.146:0/680856765 wait complete. 2024-01-21T20:05:38.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:38.191+0000 7f6196bb3700 1 Processor -- start 2024-01-21T20:05:38.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:38.191+0000 7f6196bb3700 1 -- start start 2024-01-21T20:05:38.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:38.191+0000 7f6196bb3700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f61900f5410 0x7f6190105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:38.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:38.191+0000 7f6196bb3700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6190105b20 con 0x7f61900f5410 2024-01-21T20:05:38.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:38.191+0000 7f619494f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f61900f5410 0x7f6190105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:38.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:38.192+0000 7f619494f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f61900f5410 0x7f6190105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:42942/0 (socket says 172.21.15.146:42942) 2024-01-21T20:05:38.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:38.192+0000 7f619494f700 1 -- 172.21.15.146:0/3644069373 learned_addr learned my addr 172.21.15.146:0/3644069373 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:05:38.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:38.192+0000 7f619494f700 1 -- 172.21.15.146:0/3644069373 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6184005040 con 0x7f61900f5410 2024-01-21T20:05:38.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:38.192+0000 7f619494f700 1 --2- 172.21.15.146:0/3644069373 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f61900f5410 0x7f6190105630 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f6184013510 tx=0x7f618400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:38.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:38.192+0000 7f618d7fa700 1 -- 172.21.15.146:0/3644069373 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6184009320 con 0x7f61900f5410 2024-01-21T20:05:38.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:38.192+0000 7f618d7fa700 1 -- 172.21.15.146:0/3644069373 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6184013f00 con 0x7f61900f5410 2024-01-21T20:05:38.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:38.192+0000 7f6196bb3700 1 -- 172.21.15.146:0/3644069373 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6190105d20 con 0x7f61900f5410 2024-01-21T20:05:38.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:38.192+0000 7f618d7fa700 1 -- 172.21.15.146:0/3644069373 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6184014d90 con 0x7f61900f5410 2024-01-21T20:05:38.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:38.192+0000 7f6196bb3700 1 -- 172.21.15.146:0/3644069373 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61901061b0 con 0x7f61900f5410 2024-01-21T20:05:38.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:38.193+0000 7f618d7fa700 1 -- 172.21.15.146:0/3644069373 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f61840095f0 con 0x7f61900f5410 2024-01-21T20:05:38.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:38.193+0000 7f6196bb3700 1 -- 172.21.15.146:0/3644069373 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6190045ad0 con 0x7f61900f5410 2024-01-21T20:05:38.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:38.193+0000 7f618d7fa700 1 --2- 172.21.15.146:0/3644069373 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f618002ff60 0x7f6180032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:38.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:38.193+0000 7f618d7fa700 1 -- 172.21.15.146:0/3644069373 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f61840410c0 con 0x7f61900f5410 2024-01-21T20:05:38.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:38.193+0000 7f618ffff700 1 --2- 172.21.15.146:0/3644069373 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f618002ff60 0x7f6180032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:38.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:38.194+0000 7f618ffff700 1 --2- 172.21.15.146:0/3644069373 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f618002ff60 0x7f6180032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f617c0096a0 tx=0x7f617c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:38.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:38.198+0000 7f618d7fa700 1 -- 172.21.15.146:0/3644069373 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6184014420 con 0x7f61900f5410 2024-01-21T20:05:38.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:38.402+0000 7f6196bb3700 1 -- 172.21.15.146:0/3644069373 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f61900fae60 con 0x7f61900f5410 2024-01-21T20:05:38.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:38.402+0000 7f618d7fa700 1 -- 172.21.15.146:0/3644069373 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6184010230 con 0x7f61900f5410 2024-01-21T20:05:38.392 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:05:38.393 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:05:38.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:38.404+0000 7f6196bb3700 1 -- 172.21.15.146:0/3644069373 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f618002ff60 msgr2=0x7f6180032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:38.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:38.404+0000 7f6196bb3700 1 --2- 172.21.15.146:0/3644069373 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f618002ff60 0x7f6180032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f617c0096a0 tx=0x7f617c006b40).stop 2024-01-21T20:05:38.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:38.404+0000 7f6196bb3700 1 -- 172.21.15.146:0/3644069373 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f61900f5410 msgr2=0x7f6190105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:38.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:38.404+0000 7f6196bb3700 1 --2- 172.21.15.146:0/3644069373 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f61900f5410 0x7f6190105630 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f6184013510 tx=0x7f618400b040).stop 2024-01-21T20:05:38.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:38.404+0000 7f6196bb3700 1 -- 172.21.15.146:0/3644069373 shutdown_connections 2024-01-21T20:05:38.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:38.404+0000 7f6196bb3700 1 --2- 172.21.15.146:0/3644069373 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f618002ff60 0x7f6180032420 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:38.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:38.404+0000 7f6196bb3700 1 --2- 172.21.15.146:0/3644069373 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f61900f5410 0x7f6190105630 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:38.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:38.404+0000 7f6196bb3700 1 -- 172.21.15.146:0/3644069373 >> 172.21.15.146:0/3644069373 conn(0x7f61900f0b70 msgr2=0x7f61900f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:38.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:38.404+0000 7f6196bb3700 1 -- 172.21.15.146:0/3644069373 shutdown_connections 2024-01-21T20:05:38.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:38.404+0000 7f6196bb3700 1 -- 172.21.15.146:0/3644069373 wait complete. 2024-01-21T20:05:38.395 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:05:39.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:38 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:05:37.186849+0000 mgr.smithi053.fgsmol (mgr.14164) 200 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:39.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:38 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:05:38.402934+0000 mon.smithi053 (mon.0) 264 : audit [DBG] from='client.? 172.21.15.146:0/3644069373' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:05:39.802 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:05:39.802 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:05:40.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:41.001+0000 7faae5760700 1 -- 172.21.15.146:0/2098480829 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faae0060180 msgr2=0x7faae00f8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:40.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:41.001+0000 7faae5760700 1 --2- 172.21.15.146:0/2098480829 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faae0060180 0x7faae00f8f80 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7faad0009fb0 tx=0x7faad0009b30).stop 2024-01-21T20:05:40.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:41.001+0000 7faae5760700 1 -- 172.21.15.146:0/2098480829 shutdown_connections 2024-01-21T20:05:40.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:41.001+0000 7faae5760700 1 --2- 172.21.15.146:0/2098480829 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faae0060180 0x7faae00f8f80 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:40.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:41.001+0000 7faae5760700 1 -- 172.21.15.146:0/2098480829 >> 172.21.15.146:0/2098480829 conn(0x7faae00f09a0 msgr2=0x7faae00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:40.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:41.001+0000 7faae5760700 1 -- 172.21.15.146:0/2098480829 shutdown_connections 2024-01-21T20:05:40.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:41.001+0000 7faae5760700 1 -- 172.21.15.146:0/2098480829 wait complete. 2024-01-21T20:05:40.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:41.001+0000 7faae5760700 1 Processor -- start 2024-01-21T20:05:40.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:41.002+0000 7faae5760700 1 -- start start 2024-01-21T20:05:40.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:41.002+0000 7faae5760700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faae0060180 0x7faae0106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:40.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:41.002+0000 7faae5760700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae01069f0 con 0x7faae0060180 2024-01-21T20:05:40.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:41.002+0000 7faadffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faae0060180 0x7faae0106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:40.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:41.002+0000 7faadffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faae0060180 0x7faae0106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:42946/0 (socket says 172.21.15.146:42946) 2024-01-21T20:05:40.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:41.002+0000 7faadffff700 1 -- 172.21.15.146:0/4215927461 learned_addr learned my addr 172.21.15.146:0/4215927461 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:05:40.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:41.003+0000 7faadffff700 1 -- 172.21.15.146:0/4215927461 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad0005040 con 0x7faae0060180 2024-01-21T20:05:40.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:41.003+0000 7faadffff700 1 --2- 172.21.15.146:0/4215927461 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faae0060180 0x7faae0106500 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7faad0013510 tx=0x7faad000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:40.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:41.003+0000 7faadcff9700 1 -- 172.21.15.146:0/4215927461 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faad0009320 con 0x7faae0060180 2024-01-21T20:05:40.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:41.003+0000 7faadcff9700 1 -- 172.21.15.146:0/4215927461 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faad0013f00 con 0x7faae0060180 2024-01-21T20:05:40.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:41.003+0000 7faae5760700 1 -- 172.21.15.146:0/4215927461 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faae0106bf0 con 0x7faae0060180 2024-01-21T20:05:40.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:41.003+0000 7faadcff9700 1 -- 172.21.15.146:0/4215927461 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faad0014d90 con 0x7faae0060180 2024-01-21T20:05:40.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:41.003+0000 7faae5760700 1 -- 172.21.15.146:0/4215927461 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faae0107080 con 0x7faae0060180 2024-01-21T20:05:40.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:41.004+0000 7faadcff9700 1 -- 172.21.15.146:0/4215927461 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faad00095f0 con 0x7faae0060180 2024-01-21T20:05:40.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:41.004+0000 7faadcff9700 1 --2- 172.21.15.146:0/4215927461 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7faac802ff60 0x7faac8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:40.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:41.004+0000 7faae5760700 1 -- 172.21.15.146:0/4215927461 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faae0045ad0 con 0x7faae0060180 2024-01-21T20:05:40.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:41.004+0000 7faadcff9700 1 -- 172.21.15.146:0/4215927461 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faad00410c0 con 0x7faae0060180 2024-01-21T20:05:40.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:41.004+0000 7faadf7fe700 1 --2- 172.21.15.146:0/4215927461 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7faac802ff60 0x7faac8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:40.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:41.005+0000 7faadf7fe700 1 --2- 172.21.15.146:0/4215927461 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7faac802ff60 0x7faac8032420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7faad40096a0 tx=0x7faad4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:40.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:41.009+0000 7faadcff9700 1 -- 172.21.15.146:0/4215927461 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faad006d2b0 con 0x7faae0060180 2024-01-21T20:05:41.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:40 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:05:39.187114+0000 mgr.smithi053.fgsmol (mgr.14164) 201 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:41.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:41.212+0000 7faae5760700 1 -- 172.21.15.146:0/4215927461 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faae0107f30 con 0x7faae0060180 2024-01-21T20:05:41.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:41.213+0000 7faadcff9700 1 -- 172.21.15.146:0/4215927461 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7faad001d7f0 con 0x7faae0060180 2024-01-21T20:05:41.203 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:05:41.203 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:05:41.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:41.215+0000 7faae5760700 1 -- 172.21.15.146:0/4215927461 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7faac802ff60 msgr2=0x7faac8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:41.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:41.215+0000 7faae5760700 1 --2- 172.21.15.146:0/4215927461 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7faac802ff60 0x7faac8032420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7faad40096a0 tx=0x7faad4006b40).stop 2024-01-21T20:05:41.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:41.215+0000 7faae5760700 1 -- 172.21.15.146:0/4215927461 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faae0060180 msgr2=0x7faae0106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:41.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:41.215+0000 7faae5760700 1 --2- 172.21.15.146:0/4215927461 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faae0060180 0x7faae0106500 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7faad0013510 tx=0x7faad000b040).stop 2024-01-21T20:05:41.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:41.215+0000 7faae5760700 1 -- 172.21.15.146:0/4215927461 shutdown_connections 2024-01-21T20:05:41.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:41.215+0000 7faae5760700 1 --2- 172.21.15.146:0/4215927461 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7faac802ff60 0x7faac8032420 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:41.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:41.215+0000 7faae5760700 1 --2- 172.21.15.146:0/4215927461 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faae0060180 0x7faae0106500 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:41.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:41.215+0000 7faae5760700 1 -- 172.21.15.146:0/4215927461 >> 172.21.15.146:0/4215927461 conn(0x7faae00f09a0 msgr2=0x7faae00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:41.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:41.215+0000 7faae5760700 1 -- 172.21.15.146:0/4215927461 shutdown_connections 2024-01-21T20:05:41.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:41.215+0000 7faae5760700 1 -- 172.21.15.146:0/4215927461 wait complete. 2024-01-21T20:05:41.205 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:05:42.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:05:41.213288+0000 mon.smithi053 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.146:0/4215927461' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:05:42.635 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:05:42.635 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:05:43.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:42 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:05:41.187395+0000 mgr.smithi053.fgsmol (mgr.14164) 202 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:43.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:43.867+0000 7f2635877700 1 -- 172.21.15.146:0/1505968517 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f26300f6510 msgr2=0x7f26300f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:43.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:43.867+0000 7f2635877700 1 --2- 172.21.15.146:0/1505968517 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f26300f6510 0x7f26300f6930 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f2628009fb0 tx=0x7f2628009b30).stop 2024-01-21T20:05:43.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:43.867+0000 7f2635877700 1 -- 172.21.15.146:0/1505968517 shutdown_connections 2024-01-21T20:05:43.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:43.867+0000 7f2635877700 1 --2- 172.21.15.146:0/1505968517 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f26300f6510 0x7f26300f6930 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:43.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:43.867+0000 7f2635877700 1 -- 172.21.15.146:0/1505968517 >> 172.21.15.146:0/1505968517 conn(0x7f26300f1c70 msgr2=0x7f26300f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:43.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:43.867+0000 7f2635877700 1 -- 172.21.15.146:0/1505968517 shutdown_connections 2024-01-21T20:05:43.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:43.867+0000 7f2635877700 1 -- 172.21.15.146:0/1505968517 wait complete. 2024-01-21T20:05:43.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:43.868+0000 7f2635877700 1 Processor -- start 2024-01-21T20:05:43.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:43.868+0000 7f2635877700 1 -- start start 2024-01-21T20:05:43.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:43.868+0000 7f2635877700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f26300f6510 0x7f2630106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:43.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:43.868+0000 7f2635877700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2630106c20 con 0x7f26300f6510 2024-01-21T20:05:43.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:43.868+0000 7f262effd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f26300f6510 0x7f2630106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:43.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:43.868+0000 7f262effd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f26300f6510 0x7f2630106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:42960/0 (socket says 172.21.15.146:42960) 2024-01-21T20:05:43.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:43.868+0000 7f262effd700 1 -- 172.21.15.146:0/2655461051 learned_addr learned my addr 172.21.15.146:0/2655461051 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:05:43.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:43.869+0000 7f262effd700 1 -- 172.21.15.146:0/2655461051 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2628005040 con 0x7f26300f6510 2024-01-21T20:05:43.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:43.869+0000 7f262effd700 1 --2- 172.21.15.146:0/2655461051 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f26300f6510 0x7f2630106730 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f2628013510 tx=0x7f262800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:43.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:43.869+0000 7f26177fe700 1 -- 172.21.15.146:0/2655461051 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2628009320 con 0x7f26300f6510 2024-01-21T20:05:43.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:43.869+0000 7f2635877700 1 -- 172.21.15.146:0/2655461051 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2630106e20 con 0x7f26300f6510 2024-01-21T20:05:43.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:43.869+0000 7f26177fe700 1 -- 172.21.15.146:0/2655461051 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2628013f00 con 0x7f26300f6510 2024-01-21T20:05:43.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:43.869+0000 7f26177fe700 1 -- 172.21.15.146:0/2655461051 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2628014d90 con 0x7f26300f6510 2024-01-21T20:05:43.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:43.869+0000 7f2635877700 1 -- 172.21.15.146:0/2655461051 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26301072b0 con 0x7f26300f6510 2024-01-21T20:05:43.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:43.870+0000 7f26177fe700 1 -- 172.21.15.146:0/2655461051 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f26280095f0 con 0x7f26300f6510 2024-01-21T20:05:43.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:43.870+0000 7f2635877700 1 -- 172.21.15.146:0/2655461051 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2630045ad0 con 0x7f26300f6510 2024-01-21T20:05:43.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:43.870+0000 7f26177fe700 1 --2- 172.21.15.146:0/2655461051 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f261802ff60 0x7f2618032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:43.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:43.870+0000 7f26177fe700 1 -- 172.21.15.146:0/2655461051 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f26280410a0 con 0x7f26300f6510 2024-01-21T20:05:43.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:43.870+0000 7f262e7fc700 1 --2- 172.21.15.146:0/2655461051 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f261802ff60 0x7f2618032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:43.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:43.870+0000 7f262e7fc700 1 --2- 172.21.15.146:0/2655461051 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f261802ff60 0x7f2618032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f26240096a0 tx=0x7f2624006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:43.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:43.875+0000 7f26177fe700 1 -- 172.21.15.146:0/2655461051 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f262801d7f0 con 0x7f26300f6510 2024-01-21T20:05:44.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:44.080+0000 7f2635877700 1 -- 172.21.15.146:0/2655461051 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f26300fbf60 con 0x7f26300f6510 2024-01-21T20:05:44.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:44.080+0000 7f26177fe700 1 -- 172.21.15.146:0/2655461051 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2628010230 con 0x7f26300f6510 2024-01-21T20:05:44.070 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:05:44.070 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:05:44.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:44.082+0000 7f2635877700 1 -- 172.21.15.146:0/2655461051 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f261802ff60 msgr2=0x7f2618032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:44.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:44.082+0000 7f2635877700 1 --2- 172.21.15.146:0/2655461051 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f261802ff60 0x7f2618032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f26240096a0 tx=0x7f2624006b40).stop 2024-01-21T20:05:44.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:44.082+0000 7f2635877700 1 -- 172.21.15.146:0/2655461051 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f26300f6510 msgr2=0x7f2630106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:44.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:44.082+0000 7f2635877700 1 --2- 172.21.15.146:0/2655461051 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f26300f6510 0x7f2630106730 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f2628013510 tx=0x7f262800b040).stop 2024-01-21T20:05:44.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:44.082+0000 7f2635877700 1 -- 172.21.15.146:0/2655461051 shutdown_connections 2024-01-21T20:05:44.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:44.082+0000 7f2635877700 1 --2- 172.21.15.146:0/2655461051 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f261802ff60 0x7f2618032420 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:44.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:44.082+0000 7f2635877700 1 --2- 172.21.15.146:0/2655461051 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f26300f6510 0x7f2630106730 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:44.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:44.082+0000 7f2635877700 1 -- 172.21.15.146:0/2655461051 >> 172.21.15.146:0/2655461051 conn(0x7f26300f1c70 msgr2=0x7f26300f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:44.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:44.082+0000 7f2635877700 1 -- 172.21.15.146:0/2655461051 shutdown_connections 2024-01-21T20:05:44.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:44.082+0000 7f2635877700 1 -- 172.21.15.146:0/2655461051 wait complete. 2024-01-21T20:05:44.073 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:05:45.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:44 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:05:43.187780+0000 mgr.smithi053.fgsmol (mgr.14164) 203 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:45.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:44 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:05:44.080707+0000 mon.smithi053 (mon.0) 266 : audit [DBG] from='client.? 172.21.15.146:0/2655461051' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:05:45.498 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:05:45.498 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:05:46.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:46.704+0000 7f1ac5469700 1 -- 172.21.15.146:0/2465601466 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ac00f6510 msgr2=0x7f1ac00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:46.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:46.704+0000 7f1ac5469700 1 --2- 172.21.15.146:0/2465601466 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ac00f6510 0x7f1ac00f6930 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f1ab4009fb0 tx=0x7f1ab4009b30).stop 2024-01-21T20:05:46.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:46.704+0000 7f1ac5469700 1 -- 172.21.15.146:0/2465601466 shutdown_connections 2024-01-21T20:05:46.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:46.704+0000 7f1ac5469700 1 --2- 172.21.15.146:0/2465601466 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ac00f6510 0x7f1ac00f6930 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:46.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:46.704+0000 7f1ac5469700 1 -- 172.21.15.146:0/2465601466 >> 172.21.15.146:0/2465601466 conn(0x7f1ac00f1c70 msgr2=0x7f1ac00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:46.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:46.704+0000 7f1ac5469700 1 -- 172.21.15.146:0/2465601466 shutdown_connections 2024-01-21T20:05:46.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:46.704+0000 7f1ac5469700 1 -- 172.21.15.146:0/2465601466 wait complete. 2024-01-21T20:05:46.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:46.705+0000 7f1ac5469700 1 Processor -- start 2024-01-21T20:05:46.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:46.705+0000 7f1ac5469700 1 -- start start 2024-01-21T20:05:46.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:46.705+0000 7f1ac5469700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ac00f6510 0x7f1ac0106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:46.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:46.705+0000 7f1ac5469700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac0106c10 con 0x7f1ac00f6510 2024-01-21T20:05:46.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:46.705+0000 7f1abeffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ac00f6510 0x7f1ac0106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:46.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:46.705+0000 7f1abeffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ac00f6510 0x7f1ac0106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:42980/0 (socket says 172.21.15.146:42980) 2024-01-21T20:05:46.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:46.705+0000 7f1abeffd700 1 -- 172.21.15.146:0/1739070322 learned_addr learned my addr 172.21.15.146:0/1739070322 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:05:46.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:46.706+0000 7f1abeffd700 1 -- 172.21.15.146:0/1739070322 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab4005040 con 0x7f1ac00f6510 2024-01-21T20:05:46.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:46.706+0000 7f1abeffd700 1 --2- 172.21.15.146:0/1739070322 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ac00f6510 0x7f1ac0106720 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f1ab4013510 tx=0x7f1ab400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:46.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:46.706+0000 7f1aa77fe700 1 -- 172.21.15.146:0/1739070322 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ab4009320 con 0x7f1ac00f6510 2024-01-21T20:05:46.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:46.706+0000 7f1ac5469700 1 -- 172.21.15.146:0/1739070322 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ac0106e10 con 0x7f1ac00f6510 2024-01-21T20:05:46.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:46.706+0000 7f1aa77fe700 1 -- 172.21.15.146:0/1739070322 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1ab4013f00 con 0x7f1ac00f6510 2024-01-21T20:05:46.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:46.706+0000 7f1aa77fe700 1 -- 172.21.15.146:0/1739070322 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ab4014d90 con 0x7f1ac00f6510 2024-01-21T20:05:46.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:46.706+0000 7f1ac5469700 1 -- 172.21.15.146:0/1739070322 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ac01072a0 con 0x7f1ac00f6510 2024-01-21T20:05:46.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:46.707+0000 7f1aa77fe700 1 -- 172.21.15.146:0/1739070322 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1ab4014420 con 0x7f1ac00f6510 2024-01-21T20:05:46.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:46.707+0000 7f1ac5469700 1 -- 172.21.15.146:0/1739070322 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ac0045ad0 con 0x7f1ac00f6510 2024-01-21T20:05:46.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:46.707+0000 7f1aa77fe700 1 --2- 172.21.15.146:0/1739070322 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f1aa802ffb0 0x7f1aa8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:46.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:46.707+0000 7f1aa77fe700 1 -- 172.21.15.146:0/1739070322 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1ab4040ff0 con 0x7f1ac00f6510 2024-01-21T20:05:46.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:46.707+0000 7f1abe7fc700 1 --2- 172.21.15.146:0/1739070322 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f1aa802ffb0 0x7f1aa8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:46.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:46.708+0000 7f1abe7fc700 1 --2- 172.21.15.146:0/1739070322 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f1aa802ffb0 0x7f1aa8032470 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f1ab00096a0 tx=0x7f1ab0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:46.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:46.712+0000 7f1aa77fe700 1 -- 172.21.15.146:0/1739070322 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1ab406d2b0 con 0x7f1ac00f6510 2024-01-21T20:05:46.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:46.918+0000 7f1ac5469700 1 -- 172.21.15.146:0/1739070322 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1ac0108150 con 0x7f1ac00f6510 2024-01-21T20:05:46.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:46.919+0000 7f1aa77fe700 1 -- 172.21.15.146:0/1739070322 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1ab40095f0 con 0x7f1ac00f6510 2024-01-21T20:05:46.909 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:05:46.909 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:05:46.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:46.920+0000 7f1ac5469700 1 -- 172.21.15.146:0/1739070322 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f1aa802ffb0 msgr2=0x7f1aa8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:46.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:46.920+0000 7f1ac5469700 1 --2- 172.21.15.146:0/1739070322 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f1aa802ffb0 0x7f1aa8032470 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f1ab00096a0 tx=0x7f1ab0006b40).stop 2024-01-21T20:05:46.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:46.920+0000 7f1ac5469700 1 -- 172.21.15.146:0/1739070322 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ac00f6510 msgr2=0x7f1ac0106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:46.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:46.920+0000 7f1ac5469700 1 --2- 172.21.15.146:0/1739070322 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ac00f6510 0x7f1ac0106720 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f1ab4013510 tx=0x7f1ab400b040).stop 2024-01-21T20:05:46.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:46.920+0000 7f1ac5469700 1 -- 172.21.15.146:0/1739070322 shutdown_connections 2024-01-21T20:05:46.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:46.920+0000 7f1ac5469700 1 --2- 172.21.15.146:0/1739070322 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f1aa802ffb0 0x7f1aa8032470 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:46.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:46.920+0000 7f1ac5469700 1 --2- 172.21.15.146:0/1739070322 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ac00f6510 0x7f1ac0106720 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:46.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:46.920+0000 7f1ac5469700 1 -- 172.21.15.146:0/1739070322 >> 172.21.15.146:0/1739070322 conn(0x7f1ac00f1c70 msgr2=0x7f1ac00f27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:46.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:46.920+0000 7f1ac5469700 1 -- 172.21.15.146:0/1739070322 shutdown_connections 2024-01-21T20:05:46.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:46.920+0000 7f1ac5469700 1 -- 172.21.15.146:0/1739070322 wait complete. 2024-01-21T20:05:46.911 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:05:47.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:46 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:05:45.188135+0000 mgr.smithi053.fgsmol (mgr.14164) 204 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:48.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:47 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:05:46.919079+0000 mon.smithi053 (mon.0) 267 : audit [DBG] from='client.? 172.21.15.146:0/1739070322' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:05:48.292 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:05:48.292 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:05:49.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:48 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:05:47.188404+0000 mgr.smithi053.fgsmol (mgr.14164) 205 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:49.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:49.543+0000 7fd4fa3d9700 1 -- 172.21.15.146:0/1309717718 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd4f40f5410 msgr2=0x7fd4f40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:49.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:49.543+0000 7fd4fa3d9700 1 --2- 172.21.15.146:0/1309717718 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd4f40f5410 0x7fd4f40f5830 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7fd4e4009fb0 tx=0x7fd4e4009b30).stop 2024-01-21T20:05:49.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:49.543+0000 7fd4fa3d9700 1 -- 172.21.15.146:0/1309717718 shutdown_connections 2024-01-21T20:05:49.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:49.543+0000 7fd4fa3d9700 1 --2- 172.21.15.146:0/1309717718 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd4f40f5410 0x7fd4f40f5830 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:49.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:49.543+0000 7fd4fa3d9700 1 -- 172.21.15.146:0/1309717718 >> 172.21.15.146:0/1309717718 conn(0x7fd4f40f0b70 msgr2=0x7fd4f40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:49.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:49.543+0000 7fd4fa3d9700 1 -- 172.21.15.146:0/1309717718 shutdown_connections 2024-01-21T20:05:49.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:49.543+0000 7fd4fa3d9700 1 -- 172.21.15.146:0/1309717718 wait complete. 2024-01-21T20:05:49.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:49.544+0000 7fd4fa3d9700 1 Processor -- start 2024-01-21T20:05:49.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:49.544+0000 7fd4fa3d9700 1 -- start start 2024-01-21T20:05:49.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:49.544+0000 7fd4fa3d9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd4f40f5410 0x7fd4f4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:49.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:49.544+0000 7fd4fa3d9700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f4105b20 con 0x7fd4f40f5410 2024-01-21T20:05:49.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:49.544+0000 7fd4f3fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd4f40f5410 0x7fd4f4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:49.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:49.544+0000 7fd4f3fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd4f40f5410 0x7fd4f4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:46700/0 (socket says 172.21.15.146:46700) 2024-01-21T20:05:49.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:49.544+0000 7fd4f3fff700 1 -- 172.21.15.146:0/2061270428 learned_addr learned my addr 172.21.15.146:0/2061270428 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:05:49.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:49.545+0000 7fd4f3fff700 1 -- 172.21.15.146:0/2061270428 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e4005040 con 0x7fd4f40f5410 2024-01-21T20:05:49.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:49.545+0000 7fd4f3fff700 1 --2- 172.21.15.146:0/2061270428 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd4f40f5410 0x7fd4f4105630 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fd4e4013510 tx=0x7fd4e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:49.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:49.545+0000 7fd4f0ff9700 1 -- 172.21.15.146:0/2061270428 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4e4009320 con 0x7fd4f40f5410 2024-01-21T20:05:49.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:49.545+0000 7fd4fa3d9700 1 -- 172.21.15.146:0/2061270428 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4f4105d20 con 0x7fd4f40f5410 2024-01-21T20:05:49.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:49.545+0000 7fd4f0ff9700 1 -- 172.21.15.146:0/2061270428 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd4e4013f00 con 0x7fd4f40f5410 2024-01-21T20:05:49.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:49.545+0000 7fd4f0ff9700 1 -- 172.21.15.146:0/2061270428 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4e4014d90 con 0x7fd4f40f5410 2024-01-21T20:05:49.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:49.545+0000 7fd4fa3d9700 1 -- 172.21.15.146:0/2061270428 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4f41061b0 con 0x7fd4f40f5410 2024-01-21T20:05:49.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:49.546+0000 7fd4f0ff9700 1 -- 172.21.15.146:0/2061270428 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd4e40095f0 con 0x7fd4f40f5410 2024-01-21T20:05:49.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:49.546+0000 7fd4fa3d9700 1 -- 172.21.15.146:0/2061270428 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4f4045ad0 con 0x7fd4f40f5410 2024-01-21T20:05:49.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:49.546+0000 7fd4f0ff9700 1 --2- 172.21.15.146:0/2061270428 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fd4dc02ff60 0x7fd4dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:49.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:49.546+0000 7fd4f0ff9700 1 -- 172.21.15.146:0/2061270428 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd4e40410c0 con 0x7fd4f40f5410 2024-01-21T20:05:49.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:49.546+0000 7fd4f37fe700 1 --2- 172.21.15.146:0/2061270428 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fd4dc02ff60 0x7fd4dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:49.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:49.547+0000 7fd4f37fe700 1 --2- 172.21.15.146:0/2061270428 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fd4dc02ff60 0x7fd4dc032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fd4e80096a0 tx=0x7fd4e8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:49.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:49.551+0000 7fd4f0ff9700 1 -- 172.21.15.146:0/2061270428 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd4e4014420 con 0x7fd4f40f5410 2024-01-21T20:05:49.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:49.753+0000 7fd4fa3d9700 1 -- 172.21.15.146:0/2061270428 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd4f40fae60 con 0x7fd4f40f5410 2024-01-21T20:05:49.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:49.754+0000 7fd4f0ff9700 1 -- 172.21.15.146:0/2061270428 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd4e401d7f0 con 0x7fd4f40f5410 2024-01-21T20:05:49.744 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:05:49.744 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:05:49.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:49.755+0000 7fd4fa3d9700 1 -- 172.21.15.146:0/2061270428 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fd4dc02ff60 msgr2=0x7fd4dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:49.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:49.755+0000 7fd4fa3d9700 1 --2- 172.21.15.146:0/2061270428 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fd4dc02ff60 0x7fd4dc032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fd4e80096a0 tx=0x7fd4e8006b40).stop 2024-01-21T20:05:49.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:49.755+0000 7fd4fa3d9700 1 -- 172.21.15.146:0/2061270428 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd4f40f5410 msgr2=0x7fd4f4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:49.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:49.755+0000 7fd4fa3d9700 1 --2- 172.21.15.146:0/2061270428 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd4f40f5410 0x7fd4f4105630 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fd4e4013510 tx=0x7fd4e400b040).stop 2024-01-21T20:05:49.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:49.755+0000 7fd4fa3d9700 1 -- 172.21.15.146:0/2061270428 shutdown_connections 2024-01-21T20:05:49.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:49.755+0000 7fd4fa3d9700 1 --2- 172.21.15.146:0/2061270428 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fd4dc02ff60 0x7fd4dc032420 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:49.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:49.755+0000 7fd4fa3d9700 1 --2- 172.21.15.146:0/2061270428 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd4f40f5410 0x7fd4f4105630 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:49.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:49.755+0000 7fd4fa3d9700 1 -- 172.21.15.146:0/2061270428 >> 172.21.15.146:0/2061270428 conn(0x7fd4f40f0b70 msgr2=0x7fd4f40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:49.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:49.755+0000 7fd4fa3d9700 1 -- 172.21.15.146:0/2061270428 shutdown_connections 2024-01-21T20:05:49.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:49.755+0000 7fd4fa3d9700 1 -- 172.21.15.146:0/2061270428 wait complete. 2024-01-21T20:05:49.746 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:05:50.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:49 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:05:49.754180+0000 mon.smithi053 (mon.0) 268 : audit [DBG] from='client.? 172.21.15.146:0/2061270428' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:05:51.138 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:05:51.139 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:05:51.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:50 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:05:49.188653+0000 mgr.smithi053.fgsmol (mgr.14164) 206 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:52.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:52.385+0000 7f3c13b2e700 1 -- 172.21.15.146:0/1080072410 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3c0c060280 msgr2=0x7f3c0c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:52.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:52.385+0000 7f3c13b2e700 1 --2- 172.21.15.146:0/1080072410 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3c0c060280 0x7f3c0c0f90f0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f3bfc009fb0 tx=0x7f3bfc009b30).stop 2024-01-21T20:05:52.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:52.385+0000 7f3c13b2e700 1 -- 172.21.15.146:0/1080072410 shutdown_connections 2024-01-21T20:05:52.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:52.385+0000 7f3c13b2e700 1 --2- 172.21.15.146:0/1080072410 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3c0c060280 0x7f3c0c0f90f0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:52.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:52.385+0000 7f3c13b2e700 1 -- 172.21.15.146:0/1080072410 >> 172.21.15.146:0/1080072410 conn(0x7f3c0c0f0b70 msgr2=0x7f3c0c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:52.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:52.385+0000 7f3c13b2e700 1 -- 172.21.15.146:0/1080072410 shutdown_connections 2024-01-21T20:05:52.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:52.385+0000 7f3c13b2e700 1 -- 172.21.15.146:0/1080072410 wait complete. 2024-01-21T20:05:52.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:52.386+0000 7f3c13b2e700 1 Processor -- start 2024-01-21T20:05:52.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:52.386+0000 7f3c13b2e700 1 -- start start 2024-01-21T20:05:52.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:52.386+0000 7f3c13b2e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3c0c060280 0x7f3c0c0f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:52.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:52.386+0000 7f3c13b2e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0c0f4ff0 con 0x7f3c0c060280 2024-01-21T20:05:52.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:52.386+0000 7f3c118ca700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3c0c060280 0x7f3c0c0f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:52.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:52.386+0000 7f3c118ca700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3c0c060280 0x7f3c0c0f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:46712/0 (socket says 172.21.15.146:46712) 2024-01-21T20:05:52.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:52.386+0000 7f3c118ca700 1 -- 172.21.15.146:0/1184483541 learned_addr learned my addr 172.21.15.146:0/1184483541 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:05:52.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:52.387+0000 7f3c118ca700 1 -- 172.21.15.146:0/1184483541 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bfc005040 con 0x7f3c0c060280 2024-01-21T20:05:52.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:52.387+0000 7f3c118ca700 1 --2- 172.21.15.146:0/1184483541 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3c0c060280 0x7f3c0c0f6b70 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f3bfc013510 tx=0x7f3bfc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:52.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:52.387+0000 7f3c027fc700 1 -- 172.21.15.146:0/1184483541 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3bfc009320 con 0x7f3c0c060280 2024-01-21T20:05:52.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:52.387+0000 7f3c13b2e700 1 -- 172.21.15.146:0/1184483541 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c0c0f5190 con 0x7f3c0c060280 2024-01-21T20:05:52.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:52.387+0000 7f3c027fc700 1 -- 172.21.15.146:0/1184483541 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3bfc013f00 con 0x7f3c0c060280 2024-01-21T20:05:52.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:52.387+0000 7f3c027fc700 1 -- 172.21.15.146:0/1184483541 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3bfc014dc0 con 0x7f3c0c060280 2024-01-21T20:05:52.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:52.387+0000 7f3c13b2e700 1 -- 172.21.15.146:0/1184483541 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c0c0f5620 con 0x7f3c0c060280 2024-01-21T20:05:52.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:52.388+0000 7f3c027fc700 1 -- 172.21.15.146:0/1184483541 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3bfc0095f0 con 0x7f3c0c060280 2024-01-21T20:05:52.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:52.388+0000 7f3c13b2e700 1 -- 172.21.15.146:0/1184483541 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c0c059100 con 0x7f3c0c060280 2024-01-21T20:05:52.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:52.388+0000 7f3c027fc700 1 --2- 172.21.15.146:0/1184483541 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f3bf802ff60 0x7f3bf8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:52.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:52.388+0000 7f3c027fc700 1 -- 172.21.15.146:0/1184483541 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3bfc041080 con 0x7f3c0c060280 2024-01-21T20:05:52.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:52.388+0000 7f3c110c9700 1 --2- 172.21.15.146:0/1184483541 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f3bf802ff60 0x7f3bf8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:52.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:52.389+0000 7f3c110c9700 1 --2- 172.21.15.146:0/1184483541 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f3bf802ff60 0x7f3bf8032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f3c080096a0 tx=0x7f3c08006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:52.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:52.393+0000 7f3c027fc700 1 -- 172.21.15.146:0/1184483541 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3bfc014420 con 0x7f3c0c060280 2024-01-21T20:05:52.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:52.597+0000 7f3c13b2e700 1 -- 172.21.15.146:0/1184483541 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3c0c0fbf50 con 0x7f3c0c060280 2024-01-21T20:05:52.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:52.597+0000 7f3c027fc700 1 -- 172.21.15.146:0/1184483541 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3bfc010230 con 0x7f3c0c060280 2024-01-21T20:05:52.589 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:05:52.589 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:05:52.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:52.599+0000 7f3c13b2e700 1 -- 172.21.15.146:0/1184483541 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f3bf802ff60 msgr2=0x7f3bf8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:52.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:52.599+0000 7f3c13b2e700 1 --2- 172.21.15.146:0/1184483541 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f3bf802ff60 0x7f3bf8032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f3c080096a0 tx=0x7f3c08006b40).stop 2024-01-21T20:05:52.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:52.599+0000 7f3c13b2e700 1 -- 172.21.15.146:0/1184483541 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3c0c060280 msgr2=0x7f3c0c0f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:52.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:52.599+0000 7f3c13b2e700 1 --2- 172.21.15.146:0/1184483541 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3c0c060280 0x7f3c0c0f6b70 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f3bfc013510 tx=0x7f3bfc00b040).stop 2024-01-21T20:05:52.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:52.599+0000 7f3c13b2e700 1 -- 172.21.15.146:0/1184483541 shutdown_connections 2024-01-21T20:05:52.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:52.599+0000 7f3c13b2e700 1 --2- 172.21.15.146:0/1184483541 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f3bf802ff60 0x7f3bf8032420 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:52.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:52.599+0000 7f3c13b2e700 1 --2- 172.21.15.146:0/1184483541 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3c0c060280 0x7f3c0c0f6b70 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:52.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:52.599+0000 7f3c13b2e700 1 -- 172.21.15.146:0/1184483541 >> 172.21.15.146:0/1184483541 conn(0x7f3c0c0f0b70 msgr2=0x7f3c0c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:52.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:52.599+0000 7f3c13b2e700 1 -- 172.21.15.146:0/1184483541 shutdown_connections 2024-01-21T20:05:52.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:52.599+0000 7f3c13b2e700 1 -- 172.21.15.146:0/1184483541 wait complete. 2024-01-21T20:05:52.591 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:05:53.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:52 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:05:51.188915+0000 mgr.smithi053.fgsmol (mgr.14164) 207 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:53.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:52 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:05:52.598848+0000 mon.smithi053 (mon.0) 269 : audit [DBG] from='client.? 172.21.15.146:0/1184483541' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:05:53.981 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:05:53.981 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:05:55.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:54 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:05:53.189278+0000 mgr.smithi053.fgsmol (mgr.14164) 208 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:55.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:55.241+0000 7fe015240700 1 -- 172.21.15.146:0/3326802296 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0100f5410 msgr2=0x7fe0100f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:55.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:55.241+0000 7fe015240700 1 --2- 172.21.15.146:0/3326802296 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0100f5410 0x7fe0100f5830 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7fe004009fb0 tx=0x7fe004009b30).stop 2024-01-21T20:05:55.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:55.242+0000 7fe015240700 1 -- 172.21.15.146:0/3326802296 shutdown_connections 2024-01-21T20:05:55.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:55.242+0000 7fe015240700 1 --2- 172.21.15.146:0/3326802296 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0100f5410 0x7fe0100f5830 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:55.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:55.242+0000 7fe015240700 1 -- 172.21.15.146:0/3326802296 >> 172.21.15.146:0/3326802296 conn(0x7fe0100f0b70 msgr2=0x7fe0100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:55.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:55.242+0000 7fe015240700 1 -- 172.21.15.146:0/3326802296 shutdown_connections 2024-01-21T20:05:55.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:55.242+0000 7fe015240700 1 -- 172.21.15.146:0/3326802296 wait complete. 2024-01-21T20:05:55.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:55.242+0000 7fe015240700 1 Processor -- start 2024-01-21T20:05:55.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:55.242+0000 7fe015240700 1 -- start start 2024-01-21T20:05:55.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:55.242+0000 7fe015240700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0100f5410 0x7fe010105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:55.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:55.242+0000 7fe015240700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe010105b20 con 0x7fe0100f5410 2024-01-21T20:05:55.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:55.243+0000 7fe00ed9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0100f5410 0x7fe010105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:55.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:55.243+0000 7fe00ed9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0100f5410 0x7fe010105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:46720/0 (socket says 172.21.15.146:46720) 2024-01-21T20:05:55.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:55.243+0000 7fe00ed9d700 1 -- 172.21.15.146:0/3658053127 learned_addr learned my addr 172.21.15.146:0/3658053127 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:05:55.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:55.243+0000 7fe00ed9d700 1 -- 172.21.15.146:0/3658053127 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe004005040 con 0x7fe0100f5410 2024-01-21T20:05:55.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:55.243+0000 7fe00ed9d700 1 --2- 172.21.15.146:0/3658053127 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0100f5410 0x7fe010105630 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7fe004013510 tx=0x7fe00400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:55.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:55.243+0000 7fdff77fe700 1 -- 172.21.15.146:0/3658053127 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe004009320 con 0x7fe0100f5410 2024-01-21T20:05:55.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:55.243+0000 7fdff77fe700 1 -- 172.21.15.146:0/3658053127 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe004013f00 con 0x7fe0100f5410 2024-01-21T20:05:55.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:55.243+0000 7fe015240700 1 -- 172.21.15.146:0/3658053127 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe010105d20 con 0x7fe0100f5410 2024-01-21T20:05:55.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:55.243+0000 7fdff77fe700 1 -- 172.21.15.146:0/3658053127 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe004014d90 con 0x7fe0100f5410 2024-01-21T20:05:55.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:55.243+0000 7fe015240700 1 -- 172.21.15.146:0/3658053127 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0101061b0 con 0x7fe0100f5410 2024-01-21T20:05:55.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:55.244+0000 7fe015240700 1 -- 172.21.15.146:0/3658053127 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe010045ad0 con 0x7fe0100f5410 2024-01-21T20:05:55.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:55.244+0000 7fdff77fe700 1 -- 172.21.15.146:0/3658053127 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe0040095f0 con 0x7fe0100f5410 2024-01-21T20:05:55.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:55.244+0000 7fdff77fe700 1 --2- 172.21.15.146:0/3658053127 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fdff802ff60 0x7fdff8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:55.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:55.244+0000 7fdff77fe700 1 -- 172.21.15.146:0/3658053127 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe0040410c0 con 0x7fe0100f5410 2024-01-21T20:05:55.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:55.244+0000 7fe00e59c700 1 --2- 172.21.15.146:0/3658053127 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fdff802ff60 0x7fdff8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:55.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:55.245+0000 7fe00e59c700 1 --2- 172.21.15.146:0/3658053127 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fdff802ff60 0x7fdff8032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fe0000096a0 tx=0x7fe000006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:55.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:55.249+0000 7fdff77fe700 1 -- 172.21.15.146:0/3658053127 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe00401d7f0 con 0x7fe0100f5410 2024-01-21T20:05:55.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:55.453+0000 7fe015240700 1 -- 172.21.15.146:0/3658053127 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe0100fae60 con 0x7fe0100f5410 2024-01-21T20:05:55.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:55.453+0000 7fdff77fe700 1 -- 172.21.15.146:0/3658053127 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe004010230 con 0x7fe0100f5410 2024-01-21T20:05:55.444 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:05:55.445 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:05:55.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:55.455+0000 7fe015240700 1 -- 172.21.15.146:0/3658053127 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fdff802ff60 msgr2=0x7fdff8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:55.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:55.455+0000 7fe015240700 1 --2- 172.21.15.146:0/3658053127 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fdff802ff60 0x7fdff8032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fe0000096a0 tx=0x7fe000006b40).stop 2024-01-21T20:05:55.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:55.455+0000 7fe015240700 1 -- 172.21.15.146:0/3658053127 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0100f5410 msgr2=0x7fe010105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:55.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:55.455+0000 7fe015240700 1 --2- 172.21.15.146:0/3658053127 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0100f5410 0x7fe010105630 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7fe004013510 tx=0x7fe00400b040).stop 2024-01-21T20:05:55.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:55.455+0000 7fe015240700 1 -- 172.21.15.146:0/3658053127 shutdown_connections 2024-01-21T20:05:55.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:55.455+0000 7fe015240700 1 --2- 172.21.15.146:0/3658053127 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fdff802ff60 0x7fdff8032420 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:55.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:55.455+0000 7fe015240700 1 --2- 172.21.15.146:0/3658053127 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0100f5410 0x7fe010105630 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:55.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:55.455+0000 7fe015240700 1 -- 172.21.15.146:0/3658053127 >> 172.21.15.146:0/3658053127 conn(0x7fe0100f0b70 msgr2=0x7fe0100f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:55.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:55.455+0000 7fe015240700 1 -- 172.21.15.146:0/3658053127 shutdown_connections 2024-01-21T20:05:55.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:55.455+0000 7fe015240700 1 -- 172.21.15.146:0/3658053127 wait complete. 2024-01-21T20:05:55.447 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:05:56.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:55 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:05:55.454730+0000 mon.smithi053 (mon.0) 270 : audit [DBG] from='client.? 172.21.15.146:0/3658053127' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:05:56.828 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:05:56.828 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:05:57.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:56 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:05:55.189690+0000 mgr.smithi053.fgsmol (mgr.14164) 209 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:58.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:58.087+0000 7fe3ee439700 1 -- 172.21.15.146:0/3661497001 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe3e8060280 msgr2=0x7fe3e80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:58.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:58.087+0000 7fe3ee439700 1 --2- 172.21.15.146:0/3661497001 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe3e8060280 0x7fe3e80f90f0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7fe3d8009fb0 tx=0x7fe3d8009b30).stop 2024-01-21T20:05:58.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:58.088+0000 7fe3ee439700 1 -- 172.21.15.146:0/3661497001 shutdown_connections 2024-01-21T20:05:58.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:58.088+0000 7fe3ee439700 1 --2- 172.21.15.146:0/3661497001 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe3e8060280 0x7fe3e80f90f0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:58.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:58.088+0000 7fe3ee439700 1 -- 172.21.15.146:0/3661497001 >> 172.21.15.146:0/3661497001 conn(0x7fe3e80f0b70 msgr2=0x7fe3e80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:58.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:58.088+0000 7fe3ee439700 1 -- 172.21.15.146:0/3661497001 shutdown_connections 2024-01-21T20:05:58.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:58.088+0000 7fe3ee439700 1 -- 172.21.15.146:0/3661497001 wait complete. 2024-01-21T20:05:58.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:58.088+0000 7fe3ee439700 1 Processor -- start 2024-01-21T20:05:58.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:58.088+0000 7fe3ee439700 1 -- start start 2024-01-21T20:05:58.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:58.089+0000 7fe3ee439700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe3e8060280 0x7fe3e8106690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:58.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:58.089+0000 7fe3ee439700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e8106b80 con 0x7fe3e8060280 2024-01-21T20:05:58.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:58.089+0000 7fe3e7fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe3e8060280 0x7fe3e8106690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:58.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:58.089+0000 7fe3e7fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe3e8060280 0x7fe3e8106690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:54196/0 (socket says 172.21.15.146:54196) 2024-01-21T20:05:58.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:58.089+0000 7fe3e7fff700 1 -- 172.21.15.146:0/3765773005 learned_addr learned my addr 172.21.15.146:0/3765773005 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:05:58.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:58.089+0000 7fe3e7fff700 1 -- 172.21.15.146:0/3765773005 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d8005040 con 0x7fe3e8060280 2024-01-21T20:05:58.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:58.089+0000 7fe3e7fff700 1 --2- 172.21.15.146:0/3765773005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe3e8060280 0x7fe3e8106690 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7fe3d8013510 tx=0x7fe3d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:58.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:58.089+0000 7fe3e4ff9700 1 -- 172.21.15.146:0/3765773005 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3d8009320 con 0x7fe3e8060280 2024-01-21T20:05:58.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:58.090+0000 7fe3ee439700 1 -- 172.21.15.146:0/3765773005 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3e8106d80 con 0x7fe3e8060280 2024-01-21T20:05:58.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:58.090+0000 7fe3e4ff9700 1 -- 172.21.15.146:0/3765773005 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe3d8013f00 con 0x7fe3e8060280 2024-01-21T20:05:58.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:58.090+0000 7fe3e4ff9700 1 -- 172.21.15.146:0/3765773005 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3d8014dc0 con 0x7fe3e8060280 2024-01-21T20:05:58.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:58.090+0000 7fe3ee439700 1 -- 172.21.15.146:0/3765773005 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3e8107210 con 0x7fe3e8060280 2024-01-21T20:05:58.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:58.090+0000 7fe3e4ff9700 1 -- 172.21.15.146:0/3765773005 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe3d80095f0 con 0x7fe3e8060280 2024-01-21T20:05:58.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:58.090+0000 7fe3ee439700 1 -- 172.21.15.146:0/3765773005 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3e8045ad0 con 0x7fe3e8060280 2024-01-21T20:05:58.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:58.091+0000 7fe3e4ff9700 1 --2- 172.21.15.146:0/3765773005 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fe3d002ff60 0x7fe3d0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:58.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:58.091+0000 7fe3e4ff9700 1 -- 172.21.15.146:0/3765773005 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe3d8041000 con 0x7fe3e8060280 2024-01-21T20:05:58.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:58.091+0000 7fe3e77fe700 1 --2- 172.21.15.146:0/3765773005 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fe3d002ff60 0x7fe3d0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:58.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:58.091+0000 7fe3e77fe700 1 --2- 172.21.15.146:0/3765773005 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fe3d002ff60 0x7fe3d0032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fe3dc0096a0 tx=0x7fe3dc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:58.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:58.095+0000 7fe3e4ff9700 1 -- 172.21.15.146:0/3765773005 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe3d801c790 con 0x7fe3e8060280 2024-01-21T20:05:58.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:58.300+0000 7fe3ee439700 1 -- 172.21.15.146:0/3765773005 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe3e81080c0 con 0x7fe3e8060280 2024-01-21T20:05:58.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:58.301+0000 7fe3e4ff9700 1 -- 172.21.15.146:0/3765773005 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe3d8010230 con 0x7fe3e8060280 2024-01-21T20:05:58.292 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:05:58.292 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:05:58.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:58.302+0000 7fe3ee439700 1 -- 172.21.15.146:0/3765773005 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fe3d002ff60 msgr2=0x7fe3d0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:58.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:58.302+0000 7fe3ee439700 1 --2- 172.21.15.146:0/3765773005 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fe3d002ff60 0x7fe3d0032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fe3dc0096a0 tx=0x7fe3dc006b40).stop 2024-01-21T20:05:58.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:58.302+0000 7fe3ee439700 1 -- 172.21.15.146:0/3765773005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe3e8060280 msgr2=0x7fe3e8106690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:58.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:58.302+0000 7fe3ee439700 1 --2- 172.21.15.146:0/3765773005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe3e8060280 0x7fe3e8106690 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7fe3d8013510 tx=0x7fe3d800b040).stop 2024-01-21T20:05:58.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:58.303+0000 7fe3ee439700 1 -- 172.21.15.146:0/3765773005 shutdown_connections 2024-01-21T20:05:58.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:58.303+0000 7fe3ee439700 1 --2- 172.21.15.146:0/3765773005 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fe3d002ff60 0x7fe3d0032420 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:58.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:58.303+0000 7fe3ee439700 1 --2- 172.21.15.146:0/3765773005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe3e8060280 0x7fe3e8106690 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:58.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:58.303+0000 7fe3ee439700 1 -- 172.21.15.146:0/3765773005 >> 172.21.15.146:0/3765773005 conn(0x7fe3e80f0b70 msgr2=0x7fe3e80f1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:58.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:58.303+0000 7fe3ee439700 1 -- 172.21.15.146:0/3765773005 shutdown_connections 2024-01-21T20:05:58.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:05:58.303+0000 7fe3ee439700 1 -- 172.21.15.146:0/3765773005 wait complete. 2024-01-21T20:05:58.294 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:05:59.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:58 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:05:57.190016+0000 mgr.smithi053.fgsmol (mgr.14164) 210 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:59.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:05:58 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:05:58.302459+0000 mon.smithi053 (mon.0) 271 : audit [DBG] from='client.? 172.21.15.146:0/3765773005' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:05:59.692 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:05:59.692 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:06:00.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:01.000+0000 7faa27793700 1 -- 172.21.15.146:0/3383042198 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa200f5410 msgr2=0x7faa200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:00.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:01.000+0000 7faa27793700 1 --2- 172.21.15.146:0/3383042198 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa200f5410 0x7faa200f5830 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7faa14009fb0 tx=0x7faa14009b30).stop 2024-01-21T20:06:00.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:01.000+0000 7faa27793700 1 -- 172.21.15.146:0/3383042198 shutdown_connections 2024-01-21T20:06:00.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:01.000+0000 7faa27793700 1 --2- 172.21.15.146:0/3383042198 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa200f5410 0x7faa200f5830 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:00.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:01.000+0000 7faa27793700 1 -- 172.21.15.146:0/3383042198 >> 172.21.15.146:0/3383042198 conn(0x7faa200f0b70 msgr2=0x7faa200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:00.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:01.001+0000 7faa27793700 1 -- 172.21.15.146:0/3383042198 shutdown_connections 2024-01-21T20:06:00.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:01.001+0000 7faa27793700 1 -- 172.21.15.146:0/3383042198 wait complete. 2024-01-21T20:06:00.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:01.001+0000 7faa27793700 1 Processor -- start 2024-01-21T20:06:00.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:01.001+0000 7faa27793700 1 -- start start 2024-01-21T20:06:00.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:01.001+0000 7faa27793700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa200f5410 0x7faa20105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:00.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:01.001+0000 7faa27793700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa20105b20 con 0x7faa200f5410 2024-01-21T20:06:00.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:01.001+0000 7faa2552f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa200f5410 0x7faa20105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:00.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:01.001+0000 7faa2552f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa200f5410 0x7faa20105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:54206/0 (socket says 172.21.15.146:54206) 2024-01-21T20:06:00.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:01.002+0000 7faa2552f700 1 -- 172.21.15.146:0/1519398054 learned_addr learned my addr 172.21.15.146:0/1519398054 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:06:00.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:01.002+0000 7faa2552f700 1 -- 172.21.15.146:0/1519398054 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa14005040 con 0x7faa200f5410 2024-01-21T20:06:00.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:01.002+0000 7faa2552f700 1 --2- 172.21.15.146:0/1519398054 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa200f5410 0x7faa20105630 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7faa14013510 tx=0x7faa1400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:00.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:01.002+0000 7faa11ffb700 1 -- 172.21.15.146:0/1519398054 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa14009320 con 0x7faa200f5410 2024-01-21T20:06:00.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:01.002+0000 7faa11ffb700 1 -- 172.21.15.146:0/1519398054 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faa14013f00 con 0x7faa200f5410 2024-01-21T20:06:00.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:01.002+0000 7faa11ffb700 1 -- 172.21.15.146:0/1519398054 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa14014d90 con 0x7faa200f5410 2024-01-21T20:06:00.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:01.002+0000 7faa27793700 1 -- 172.21.15.146:0/1519398054 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa20105d20 con 0x7faa200f5410 2024-01-21T20:06:00.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:01.002+0000 7faa27793700 1 -- 172.21.15.146:0/1519398054 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa201061b0 con 0x7faa200f5410 2024-01-21T20:06:00.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:01.003+0000 7faa11ffb700 1 -- 172.21.15.146:0/1519398054 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faa140095f0 con 0x7faa200f5410 2024-01-21T20:06:00.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:01.003+0000 7faa27793700 1 -- 172.21.15.146:0/1519398054 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa20045ad0 con 0x7faa200f5410 2024-01-21T20:06:00.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:01.003+0000 7faa11ffb700 1 --2- 172.21.15.146:0/1519398054 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7faa0c02ff60 0x7faa0c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:00.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:01.003+0000 7faa11ffb700 1 -- 172.21.15.146:0/1519398054 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faa140410c0 con 0x7faa200f5410 2024-01-21T20:06:00.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:01.004+0000 7faa24d2e700 1 --2- 172.21.15.146:0/1519398054 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7faa0c02ff60 0x7faa0c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:00.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:01.004+0000 7faa24d2e700 1 --2- 172.21.15.146:0/1519398054 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7faa0c02ff60 0x7faa0c032420 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7faa1c0096a0 tx=0x7faa1c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:00.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:01.008+0000 7faa11ffb700 1 -- 172.21.15.146:0/1519398054 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faa1401d7f0 con 0x7faa200f5410 2024-01-21T20:06:01.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:00 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:05:59.190401+0000 mgr.smithi053.fgsmol (mgr.14164) 211 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:01.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:01.212+0000 7faa27793700 1 -- 172.21.15.146:0/1519398054 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faa200fae60 con 0x7faa200f5410 2024-01-21T20:06:01.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:01.213+0000 7faa11ffb700 1 -- 172.21.15.146:0/1519398054 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7faa14010230 con 0x7faa200f5410 2024-01-21T20:06:01.204 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:06:01.204 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:06:01.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:01.215+0000 7faa27793700 1 -- 172.21.15.146:0/1519398054 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7faa0c02ff60 msgr2=0x7faa0c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:01.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:01.215+0000 7faa27793700 1 --2- 172.21.15.146:0/1519398054 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7faa0c02ff60 0x7faa0c032420 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7faa1c0096a0 tx=0x7faa1c006b40).stop 2024-01-21T20:06:01.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:01.215+0000 7faa27793700 1 -- 172.21.15.146:0/1519398054 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa200f5410 msgr2=0x7faa20105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:01.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:01.215+0000 7faa27793700 1 --2- 172.21.15.146:0/1519398054 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa200f5410 0x7faa20105630 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7faa14013510 tx=0x7faa1400b040).stop 2024-01-21T20:06:01.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:01.215+0000 7faa27793700 1 -- 172.21.15.146:0/1519398054 shutdown_connections 2024-01-21T20:06:01.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:01.215+0000 7faa27793700 1 --2- 172.21.15.146:0/1519398054 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7faa0c02ff60 0x7faa0c032420 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:01.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:01.215+0000 7faa27793700 1 --2- 172.21.15.146:0/1519398054 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa200f5410 0x7faa20105630 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:01.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:01.215+0000 7faa27793700 1 -- 172.21.15.146:0/1519398054 >> 172.21.15.146:0/1519398054 conn(0x7faa200f0b70 msgr2=0x7faa200f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:01.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:01.215+0000 7faa27793700 1 -- 172.21.15.146:0/1519398054 shutdown_connections 2024-01-21T20:06:01.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:01.215+0000 7faa27793700 1 -- 172.21.15.146:0/1519398054 wait complete. 2024-01-21T20:06:01.207 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:06:02.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:01 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:06:01.214399+0000 mon.smithi053 (mon.0) 272 : audit [DBG] from='client.? 172.21.15.146:0/1519398054' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:06:02.642 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:06:02.643 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:06:03.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:02 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:06:01.190713+0000 mgr.smithi053.fgsmol (mgr.14164) 212 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:03.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:03.885+0000 7f1c0de71700 1 -- 172.21.15.146:0/1069386805 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c08060120 msgr2=0x7f1c080f8fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:03.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:03.885+0000 7f1c0de71700 1 --2- 172.21.15.146:0/1069386805 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c08060120 0x7f1c080f8fe0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f1bfc009fb0 tx=0x7f1bfc009b30).stop 2024-01-21T20:06:03.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:03.885+0000 7f1c0de71700 1 -- 172.21.15.146:0/1069386805 shutdown_connections 2024-01-21T20:06:03.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:03.885+0000 7f1c0de71700 1 --2- 172.21.15.146:0/1069386805 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c08060120 0x7f1c080f8fe0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:03.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:03.885+0000 7f1c0de71700 1 -- 172.21.15.146:0/1069386805 >> 172.21.15.146:0/1069386805 conn(0x7f1c080f09a0 msgr2=0x7f1c080f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:03.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:03.885+0000 7f1c0de71700 1 -- 172.21.15.146:0/1069386805 shutdown_connections 2024-01-21T20:06:03.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:03.885+0000 7f1c0de71700 1 -- 172.21.15.146:0/1069386805 wait complete. 2024-01-21T20:06:03.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:03.885+0000 7f1c0de71700 1 Processor -- start 2024-01-21T20:06:03.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:03.885+0000 7f1c0de71700 1 -- start start 2024-01-21T20:06:03.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:03.886+0000 7f1c0de71700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c08060120 0x7f1c080f6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:03.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:03.886+0000 7f1c0de71700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c080f4ee0 con 0x7f1c08060120 2024-01-21T20:06:03.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:03.886+0000 7f1c0ce6f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c08060120 0x7f1c080f6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:03.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:03.886+0000 7f1c0ce6f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c08060120 0x7f1c080f6a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:54224/0 (socket says 172.21.15.146:54224) 2024-01-21T20:06:03.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:03.886+0000 7f1c0ce6f700 1 -- 172.21.15.146:0/2166944874 learned_addr learned my addr 172.21.15.146:0/2166944874 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:06:03.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:03.886+0000 7f1c0ce6f700 1 -- 172.21.15.146:0/2166944874 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bfc005040 con 0x7f1c08060120 2024-01-21T20:06:03.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:03.886+0000 7f1c0ce6f700 1 --2- 172.21.15.146:0/2166944874 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c08060120 0x7f1c080f6a60 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f1bfc013510 tx=0x7f1bfc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:03.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:03.887+0000 7f1c057fa700 1 -- 172.21.15.146:0/2166944874 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1bfc009320 con 0x7f1c08060120 2024-01-21T20:06:03.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:03.887+0000 7f1c0de71700 1 -- 172.21.15.146:0/2166944874 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c080f5080 con 0x7f1c08060120 2024-01-21T20:06:03.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:03.887+0000 7f1c057fa700 1 -- 172.21.15.146:0/2166944874 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1bfc013f00 con 0x7f1c08060120 2024-01-21T20:06:03.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:03.887+0000 7f1c057fa700 1 -- 172.21.15.146:0/2166944874 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1bfc014d90 con 0x7f1c08060120 2024-01-21T20:06:03.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:03.887+0000 7f1c0de71700 1 -- 172.21.15.146:0/2166944874 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c080f5510 con 0x7f1c08060120 2024-01-21T20:06:03.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:03.887+0000 7f1c057fa700 1 -- 172.21.15.146:0/2166944874 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1bfc014420 con 0x7f1c08060120 2024-01-21T20:06:03.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:03.887+0000 7f1c0de71700 1 -- 172.21.15.146:0/2166944874 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c08059100 con 0x7f1c08060120 2024-01-21T20:06:03.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:03.888+0000 7f1c057fa700 1 --2- 172.21.15.146:0/2166944874 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f1bf802ff60 0x7f1bf8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:03.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:03.888+0000 7f1c057fa700 1 -- 172.21.15.146:0/2166944874 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1bfc041080 con 0x7f1c08060120 2024-01-21T20:06:03.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:03.888+0000 7f1c07fff700 1 --2- 172.21.15.146:0/2166944874 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f1bf802ff60 0x7f1bf8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:03.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:03.888+0000 7f1c07fff700 1 --2- 172.21.15.146:0/2166944874 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f1bf802ff60 0x7f1bf8032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f1bf40096a0 tx=0x7f1bf4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:03.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:03.893+0000 7f1c057fa700 1 -- 172.21.15.146:0/2166944874 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1bfc0095f0 con 0x7f1c08060120 2024-01-21T20:06:04.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:04.096+0000 7f1c0de71700 1 -- 172.21.15.146:0/2166944874 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1c080fbce0 con 0x7f1c08060120 2024-01-21T20:06:04.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:04.097+0000 7f1c057fa700 1 -- 172.21.15.146:0/2166944874 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1bfc01db70 con 0x7f1c08060120 2024-01-21T20:06:04.087 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:06:04.088 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:06:04.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:04.098+0000 7f1c0de71700 1 -- 172.21.15.146:0/2166944874 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f1bf802ff60 msgr2=0x7f1bf8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:04.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:04.098+0000 7f1c0de71700 1 --2- 172.21.15.146:0/2166944874 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f1bf802ff60 0x7f1bf8032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f1bf40096a0 tx=0x7f1bf4006b40).stop 2024-01-21T20:06:04.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:04.098+0000 7f1c0de71700 1 -- 172.21.15.146:0/2166944874 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c08060120 msgr2=0x7f1c080f6a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:04.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:04.098+0000 7f1c0de71700 1 --2- 172.21.15.146:0/2166944874 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c08060120 0x7f1c080f6a60 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f1bfc013510 tx=0x7f1bfc00b040).stop 2024-01-21T20:06:04.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:04.098+0000 7f1c0de71700 1 -- 172.21.15.146:0/2166944874 shutdown_connections 2024-01-21T20:06:04.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:04.098+0000 7f1c0de71700 1 --2- 172.21.15.146:0/2166944874 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f1bf802ff60 0x7f1bf8032420 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:04.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:04.098+0000 7f1c0de71700 1 --2- 172.21.15.146:0/2166944874 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c08060120 0x7f1c080f6a60 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:04.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:04.098+0000 7f1c0de71700 1 -- 172.21.15.146:0/2166944874 >> 172.21.15.146:0/2166944874 conn(0x7f1c080f09a0 msgr2=0x7f1c080f1450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:04.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:04.098+0000 7f1c0de71700 1 -- 172.21.15.146:0/2166944874 shutdown_connections 2024-01-21T20:06:04.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:04.098+0000 7f1c0de71700 1 -- 172.21.15.146:0/2166944874 wait complete. 2024-01-21T20:06:04.090 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:06:05.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:04 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:06:03.191010+0000 mgr.smithi053.fgsmol (mgr.14164) 213 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:05.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:04 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:06:04.097978+0000 mon.smithi053 (mon.0) 273 : audit [DBG] from='client.? 172.21.15.146:0/2166944874' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:06:05.570 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:06:05.570 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:06:06.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:06.750+0000 7f4b6a4a6700 1 -- 172.21.15.146:0/4242757086 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b640f6510 msgr2=0x7f4b640f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:06.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:06.750+0000 7f4b6a4a6700 1 --2- 172.21.15.146:0/4242757086 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b640f6510 0x7f4b640f6930 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f4b54009fb0 tx=0x7f4b54009b30).stop 2024-01-21T20:06:06.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:06.750+0000 7f4b6a4a6700 1 -- 172.21.15.146:0/4242757086 shutdown_connections 2024-01-21T20:06:06.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:06.750+0000 7f4b6a4a6700 1 --2- 172.21.15.146:0/4242757086 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b640f6510 0x7f4b640f6930 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:06.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:06.750+0000 7f4b6a4a6700 1 -- 172.21.15.146:0/4242757086 >> 172.21.15.146:0/4242757086 conn(0x7f4b640f1c70 msgr2=0x7f4b640f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:06.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:06.750+0000 7f4b6a4a6700 1 -- 172.21.15.146:0/4242757086 shutdown_connections 2024-01-21T20:06:06.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:06.750+0000 7f4b6a4a6700 1 -- 172.21.15.146:0/4242757086 wait complete. 2024-01-21T20:06:06.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:06.751+0000 7f4b6a4a6700 1 Processor -- start 2024-01-21T20:06:06.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:06.751+0000 7f4b6a4a6700 1 -- start start 2024-01-21T20:06:06.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:06.751+0000 7f4b6a4a6700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b640f6510 0x7f4b64106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:06.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:06.751+0000 7f4b6a4a6700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b64106c20 con 0x7f4b640f6510 2024-01-21T20:06:06.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:06.751+0000 7f4b63fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b640f6510 0x7f4b64106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:06.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:06.751+0000 7f4b63fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b640f6510 0x7f4b64106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:54232/0 (socket says 172.21.15.146:54232) 2024-01-21T20:06:06.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:06.751+0000 7f4b63fff700 1 -- 172.21.15.146:0/3592854058 learned_addr learned my addr 172.21.15.146:0/3592854058 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:06:06.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:06.752+0000 7f4b63fff700 1 -- 172.21.15.146:0/3592854058 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b54005040 con 0x7f4b640f6510 2024-01-21T20:06:06.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:06.752+0000 7f4b63fff700 1 --2- 172.21.15.146:0/3592854058 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b640f6510 0x7f4b64106730 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f4b54013510 tx=0x7f4b5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:06.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:06.752+0000 7f4b60ff9700 1 -- 172.21.15.146:0/3592854058 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b54009320 con 0x7f4b640f6510 2024-01-21T20:06:06.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:06.752+0000 7f4b60ff9700 1 -- 172.21.15.146:0/3592854058 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4b54013f00 con 0x7f4b640f6510 2024-01-21T20:06:06.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:06.752+0000 7f4b6a4a6700 1 -- 172.21.15.146:0/3592854058 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b64106e20 con 0x7f4b640f6510 2024-01-21T20:06:06.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:06.752+0000 7f4b60ff9700 1 -- 172.21.15.146:0/3592854058 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b54014d90 con 0x7f4b640f6510 2024-01-21T20:06:06.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:06.752+0000 7f4b6a4a6700 1 -- 172.21.15.146:0/3592854058 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b641072b0 con 0x7f4b640f6510 2024-01-21T20:06:06.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:06.753+0000 7f4b60ff9700 1 -- 172.21.15.146:0/3592854058 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4b540095f0 con 0x7f4b640f6510 2024-01-21T20:06:06.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:06.753+0000 7f4b6a4a6700 1 -- 172.21.15.146:0/3592854058 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b64045ad0 con 0x7f4b640f6510 2024-01-21T20:06:06.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:06.753+0000 7f4b60ff9700 1 --2- 172.21.15.146:0/3592854058 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f4b4c02ff60 0x7f4b4c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:06.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:06.753+0000 7f4b60ff9700 1 -- 172.21.15.146:0/3592854058 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4b54041000 con 0x7f4b640f6510 2024-01-21T20:06:06.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:06.753+0000 7f4b637fe700 1 --2- 172.21.15.146:0/3592854058 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f4b4c02ff60 0x7f4b4c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:06.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:06.754+0000 7f4b637fe700 1 --2- 172.21.15.146:0/3592854058 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f4b4c02ff60 0x7f4b4c032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f4b580096a0 tx=0x7f4b58006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:06.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:06.758+0000 7f4b60ff9700 1 -- 172.21.15.146:0/3592854058 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4b54014420 con 0x7f4b640f6510 2024-01-21T20:06:06.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:06.962+0000 7f4b6a4a6700 1 -- 172.21.15.146:0/3592854058 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4b640fbf60 con 0x7f4b640f6510 2024-01-21T20:06:06.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:06.963+0000 7f4b60ff9700 1 -- 172.21.15.146:0/3592854058 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4b5401d7f0 con 0x7f4b640f6510 2024-01-21T20:06:06.954 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:06:06.954 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:06:06.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:06.965+0000 7f4b6a4a6700 1 -- 172.21.15.146:0/3592854058 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f4b4c02ff60 msgr2=0x7f4b4c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:06.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:06.965+0000 7f4b6a4a6700 1 --2- 172.21.15.146:0/3592854058 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f4b4c02ff60 0x7f4b4c032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f4b580096a0 tx=0x7f4b58006b40).stop 2024-01-21T20:06:06.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:06.965+0000 7f4b6a4a6700 1 -- 172.21.15.146:0/3592854058 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b640f6510 msgr2=0x7f4b64106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:06.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:06.965+0000 7f4b6a4a6700 1 --2- 172.21.15.146:0/3592854058 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b640f6510 0x7f4b64106730 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f4b54013510 tx=0x7f4b5400b040).stop 2024-01-21T20:06:06.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:06.965+0000 7f4b6a4a6700 1 -- 172.21.15.146:0/3592854058 shutdown_connections 2024-01-21T20:06:06.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:06.965+0000 7f4b6a4a6700 1 --2- 172.21.15.146:0/3592854058 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f4b4c02ff60 0x7f4b4c032420 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:06.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:06.965+0000 7f4b6a4a6700 1 --2- 172.21.15.146:0/3592854058 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b640f6510 0x7f4b64106730 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:06.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:06.965+0000 7f4b6a4a6700 1 -- 172.21.15.146:0/3592854058 >> 172.21.15.146:0/3592854058 conn(0x7f4b640f1c70 msgr2=0x7f4b640f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:06.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:06.965+0000 7f4b6a4a6700 1 -- 172.21.15.146:0/3592854058 shutdown_connections 2024-01-21T20:06:06.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:06.965+0000 7f4b6a4a6700 1 -- 172.21.15.146:0/3592854058 wait complete. 2024-01-21T20:06:06.957 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:06:07.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:06 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:06:05.191376+0000 mgr.smithi053.fgsmol (mgr.14164) 214 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:08.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:07 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:06:06.964559+0000 mon.smithi053 (mon.0) 274 : audit [DBG] from='client.? 172.21.15.146:0/3592854058' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:06:08.351 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:06:08.351 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:06:09.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:08 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:06:07.191762+0000 mgr.smithi053.fgsmol (mgr.14164) 215 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:09.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:09.565+0000 7f8133fff700 1 -- 172.21.15.146:0/3101940740 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81340f79a0 msgr2=0x7f81340f7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:09.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:09.565+0000 7f8133fff700 1 --2- 172.21.15.146:0/3101940740 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81340f79a0 0x7f81340f7dc0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f8124009fb0 tx=0x7f8124009b30).stop 2024-01-21T20:06:09.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:09.565+0000 7f8133fff700 1 -- 172.21.15.146:0/3101940740 shutdown_connections 2024-01-21T20:06:09.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:09.565+0000 7f8133fff700 1 --2- 172.21.15.146:0/3101940740 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81340f79a0 0x7f81340f7dc0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:09.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:09.565+0000 7f8133fff700 1 -- 172.21.15.146:0/3101940740 >> 172.21.15.146:0/3101940740 conn(0x7f81340f0120 msgr2=0x7f81340f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:09.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:09.566+0000 7f8133fff700 1 -- 172.21.15.146:0/3101940740 shutdown_connections 2024-01-21T20:06:09.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:09.566+0000 7f8133fff700 1 -- 172.21.15.146:0/3101940740 wait complete. 2024-01-21T20:06:09.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:09.566+0000 7f8133fff700 1 Processor -- start 2024-01-21T20:06:09.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:09.566+0000 7f8133fff700 1 -- start start 2024-01-21T20:06:09.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:09.566+0000 7f8133fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81340f7230 0x7f81340f56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:09.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:09.566+0000 7f8133fff700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81340f7650 con 0x7f81340f7230 2024-01-21T20:06:09.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:09.567+0000 7f8132ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81340f7230 0x7f81340f56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:09.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:09.567+0000 7f8132ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81340f7230 0x7f81340f56f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:60090/0 (socket says 172.21.15.146:60090) 2024-01-21T20:06:09.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:09.567+0000 7f8132ffd700 1 -- 172.21.15.146:0/2074343046 learned_addr learned my addr 172.21.15.146:0/2074343046 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:06:09.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:09.567+0000 7f8132ffd700 1 -- 172.21.15.146:0/2074343046 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8124005040 con 0x7f81340f7230 2024-01-21T20:06:09.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:09.567+0000 7f8132ffd700 1 --2- 172.21.15.146:0/2074343046 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81340f7230 0x7f81340f56f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f8124009f80 tx=0x7f8124009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:09.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:09.567+0000 7f811b7fe700 1 -- 172.21.15.146:0/2074343046 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81240136c0 con 0x7f81340f7230 2024-01-21T20:06:09.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:09.567+0000 7f8133fff700 1 -- 172.21.15.146:0/2074343046 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81340f5be0 con 0x7f81340f7230 2024-01-21T20:06:09.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:09.567+0000 7f811b7fe700 1 -- 172.21.15.146:0/2074343046 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f81240095f0 con 0x7f81340f7230 2024-01-21T20:06:09.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:09.567+0000 7f811b7fe700 1 -- 172.21.15.146:0/2074343046 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8124014420 con 0x7f81340f7230 2024-01-21T20:06:09.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:09.567+0000 7f8133fff700 1 -- 172.21.15.146:0/2074343046 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81340f60d0 con 0x7f81340f7230 2024-01-21T20:06:09.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:09.568+0000 7f811b7fe700 1 -- 172.21.15.146:0/2074343046 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8124025060 con 0x7f81340f7230 2024-01-21T20:06:09.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:09.568+0000 7f8133fff700 1 -- 172.21.15.146:0/2074343046 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8134059100 con 0x7f81340f7230 2024-01-21T20:06:09.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:09.568+0000 7f811b7fe700 1 --2- 172.21.15.146:0/2074343046 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f811c02ff60 0x7f811c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:09.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:09.568+0000 7f811b7fe700 1 -- 172.21.15.146:0/2074343046 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8124040d80 con 0x7f81340f7230 2024-01-21T20:06:09.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:09.568+0000 7f81327fc700 1 --2- 172.21.15.146:0/2074343046 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f811c02ff60 0x7f811c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:09.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:09.569+0000 7f81327fc700 1 --2- 172.21.15.146:0/2074343046 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f811c02ff60 0x7f811c032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f81280096a0 tx=0x7f8128006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:09.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:09.573+0000 7f811b7fe700 1 -- 172.21.15.146:0/2074343046 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f812401c950 con 0x7f81340f7230 2024-01-21T20:06:09.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:09.779+0000 7f8133fff700 1 -- 172.21.15.146:0/2074343046 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f81340f64c0 con 0x7f81340f7230 2024-01-21T20:06:09.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:09.779+0000 7f811b7fe700 1 -- 172.21.15.146:0/2074343046 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8124042090 con 0x7f81340f7230 2024-01-21T20:06:09.771 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:06:09.771 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:06:09.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:09.781+0000 7f8133fff700 1 -- 172.21.15.146:0/2074343046 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f811c02ff60 msgr2=0x7f811c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:09.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:09.781+0000 7f8133fff700 1 --2- 172.21.15.146:0/2074343046 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f811c02ff60 0x7f811c032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f81280096a0 tx=0x7f8128006b40).stop 2024-01-21T20:06:09.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:09.781+0000 7f8133fff700 1 -- 172.21.15.146:0/2074343046 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81340f7230 msgr2=0x7f81340f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:09.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:09.781+0000 7f8133fff700 1 --2- 172.21.15.146:0/2074343046 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81340f7230 0x7f81340f56f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f8124009f80 tx=0x7f8124009b30).stop 2024-01-21T20:06:09.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:09.781+0000 7f8133fff700 1 -- 172.21.15.146:0/2074343046 shutdown_connections 2024-01-21T20:06:09.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:09.781+0000 7f8133fff700 1 --2- 172.21.15.146:0/2074343046 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f811c02ff60 0x7f811c032420 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:09.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:09.781+0000 7f8133fff700 1 --2- 172.21.15.146:0/2074343046 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81340f7230 0x7f81340f56f0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:09.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:09.781+0000 7f8133fff700 1 -- 172.21.15.146:0/2074343046 >> 172.21.15.146:0/2074343046 conn(0x7f81340f0120 msgr2=0x7f81340f0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:09.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:09.781+0000 7f8133fff700 1 -- 172.21.15.146:0/2074343046 shutdown_connections 2024-01-21T20:06:09.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:09.781+0000 7f8133fff700 1 -- 172.21.15.146:0/2074343046 wait complete. 2024-01-21T20:06:09.779 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:06:10.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:09 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:06:09.269792+0000 mon.smithi053 (mon.0) 275 : audit [INF] from='mgr.14164 172.21.15.53:0/2331671680' entity='mgr.smithi053.fgsmol' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fgsmol/mirror_snapshot_schedule"}]: dispatch 2024-01-21T20:06:10.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:09 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:06:09.278123+0000 mon.smithi053 (mon.0) 276 : audit [INF] from='mgr.14164 172.21.15.53:0/2331671680' entity='mgr.smithi053.fgsmol' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fgsmol/trash_purge_schedule"}]: dispatch 2024-01-21T20:06:10.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:09 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:06:09.780723+0000 mon.smithi053 (mon.0) 277 : audit [DBG] from='client.? 172.21.15.146:0/2074343046' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:06:11.173 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:06:11.173 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:06:11.182 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:10 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:06:09.192092+0000 mgr.smithi053.fgsmol (mgr.14164) 216 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:12.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:12.434+0000 7f8ce2e16700 1 -- 172.21.15.146:0/1949897602 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8cdc0f5410 msgr2=0x7f8cdc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:12.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:12.434+0000 7f8ce2e16700 1 --2- 172.21.15.146:0/1949897602 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8cdc0f5410 0x7f8cdc0f5830 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f8cd0009fb0 tx=0x7f8cd0009b30).stop 2024-01-21T20:06:12.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:12.434+0000 7f8ce2e16700 1 -- 172.21.15.146:0/1949897602 shutdown_connections 2024-01-21T20:06:12.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:12.434+0000 7f8ce2e16700 1 --2- 172.21.15.146:0/1949897602 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8cdc0f5410 0x7f8cdc0f5830 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:12.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:12.434+0000 7f8ce2e16700 1 -- 172.21.15.146:0/1949897602 >> 172.21.15.146:0/1949897602 conn(0x7f8cdc0f0b70 msgr2=0x7f8cdc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:12.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:12.434+0000 7f8ce2e16700 1 -- 172.21.15.146:0/1949897602 shutdown_connections 2024-01-21T20:06:12.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:12.434+0000 7f8ce2e16700 1 -- 172.21.15.146:0/1949897602 wait complete. 2024-01-21T20:06:12.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:12.435+0000 7f8ce2e16700 1 Processor -- start 2024-01-21T20:06:12.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:12.435+0000 7f8ce2e16700 1 -- start start 2024-01-21T20:06:12.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:12.435+0000 7f8ce2e16700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8cdc0f5410 0x7f8cdc10b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:12.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:12.435+0000 7f8ce2e16700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cdc107210 con 0x7f8cdc0f5410 2024-01-21T20:06:12.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:12.436+0000 7f8ce0bb2700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8cdc0f5410 0x7f8cdc10b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:12.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:12.436+0000 7f8ce0bb2700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8cdc0f5410 0x7f8cdc10b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:60102/0 (socket says 172.21.15.146:60102) 2024-01-21T20:06:12.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:12.436+0000 7f8ce0bb2700 1 -- 172.21.15.146:0/3210217734 learned_addr learned my addr 172.21.15.146:0/3210217734 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:06:12.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:12.436+0000 7f8ce0bb2700 1 -- 172.21.15.146:0/3210217734 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd0005040 con 0x7f8cdc0f5410 2024-01-21T20:06:12.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:12.436+0000 7f8ce0bb2700 1 --2- 172.21.15.146:0/3210217734 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8cdc0f5410 0x7f8cdc10b820 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f8cd0013510 tx=0x7f8cd000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:12.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:12.436+0000 7f8cd97fa700 1 -- 172.21.15.146:0/3210217734 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8cd0009320 con 0x7f8cdc0f5410 2024-01-21T20:06:12.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:12.436+0000 7f8ce2e16700 1 -- 172.21.15.146:0/3210217734 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cdc060220 con 0x7f8cdc0f5410 2024-01-21T20:06:12.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:12.436+0000 7f8cd97fa700 1 -- 172.21.15.146:0/3210217734 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8cd0013f00 con 0x7f8cdc0f5410 2024-01-21T20:06:12.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:12.436+0000 7f8cd97fa700 1 -- 172.21.15.146:0/3210217734 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8cd0014d90 con 0x7f8cdc0f5410 2024-01-21T20:06:12.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:12.436+0000 7f8ce2e16700 1 -- 172.21.15.146:0/3210217734 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cdc10bd10 con 0x7f8cdc0f5410 2024-01-21T20:06:12.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:12.437+0000 7f8cd97fa700 1 -- 172.21.15.146:0/3210217734 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8cd0014420 con 0x7f8cdc0f5410 2024-01-21T20:06:12.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:12.437+0000 7f8ce2e16700 1 -- 172.21.15.146:0/3210217734 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cdc045ad0 con 0x7f8cdc0f5410 2024-01-21T20:06:12.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:12.437+0000 7f8cd97fa700 1 --2- 172.21.15.146:0/3210217734 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f8ccc02ff60 0x7f8ccc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:12.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:12.437+0000 7f8cd97fa700 1 -- 172.21.15.146:0/3210217734 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8cd0041000 con 0x7f8cdc0f5410 2024-01-21T20:06:12.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:12.438+0000 7f8cdbfff700 1 --2- 172.21.15.146:0/3210217734 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f8ccc02ff60 0x7f8ccc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:12.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:12.438+0000 7f8cdbfff700 1 --2- 172.21.15.146:0/3210217734 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f8ccc02ff60 0x7f8ccc032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f8cc80096a0 tx=0x7f8cc8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:12.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:12.442+0000 7f8cd97fa700 1 -- 172.21.15.146:0/3210217734 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8cd00095f0 con 0x7f8cdc0f5410 2024-01-21T20:06:12.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:12.647+0000 7f8ce2e16700 1 -- 172.21.15.146:0/3210217734 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8cdc0fae60 con 0x7f8cdc0f5410 2024-01-21T20:06:12.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:12.648+0000 7f8cd97fa700 1 -- 172.21.15.146:0/3210217734 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8cd001db70 con 0x7f8cdc0f5410 2024-01-21T20:06:12.639 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:06:12.639 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:06:12.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:12.649+0000 7f8ce2e16700 1 -- 172.21.15.146:0/3210217734 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f8ccc02ff60 msgr2=0x7f8ccc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:12.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:12.649+0000 7f8ce2e16700 1 --2- 172.21.15.146:0/3210217734 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f8ccc02ff60 0x7f8ccc032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f8cc80096a0 tx=0x7f8cc8006b40).stop 2024-01-21T20:06:12.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:12.649+0000 7f8ce2e16700 1 -- 172.21.15.146:0/3210217734 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8cdc0f5410 msgr2=0x7f8cdc10b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:12.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:12.649+0000 7f8ce2e16700 1 --2- 172.21.15.146:0/3210217734 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8cdc0f5410 0x7f8cdc10b820 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f8cd0013510 tx=0x7f8cd000b040).stop 2024-01-21T20:06:12.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:12.650+0000 7f8ce2e16700 1 -- 172.21.15.146:0/3210217734 shutdown_connections 2024-01-21T20:06:12.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:12.650+0000 7f8ce2e16700 1 --2- 172.21.15.146:0/3210217734 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f8ccc02ff60 0x7f8ccc032420 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:12.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:12.650+0000 7f8ce2e16700 1 --2- 172.21.15.146:0/3210217734 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8cdc0f5410 0x7f8cdc10b820 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:12.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:12.650+0000 7f8ce2e16700 1 -- 172.21.15.146:0/3210217734 >> 172.21.15.146:0/3210217734 conn(0x7f8cdc0f0b70 msgr2=0x7f8cdc0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:12.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:12.650+0000 7f8ce2e16700 1 -- 172.21.15.146:0/3210217734 shutdown_connections 2024-01-21T20:06:12.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:12.650+0000 7f8ce2e16700 1 -- 172.21.15.146:0/3210217734 wait complete. 2024-01-21T20:06:12.641 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:06:13.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:12 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:06:11.192474+0000 mgr.smithi053.fgsmol (mgr.14164) 217 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:13.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:12 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:06:12.649298+0000 mon.smithi053 (mon.0) 278 : audit [DBG] from='client.? 172.21.15.146:0/3210217734' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:06:14.063 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:06:14.063 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:06:15.182 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:14 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:06:13.192710+0000 mgr.smithi053.fgsmol (mgr.14164) 218 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:15.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:15.238+0000 7f44ed9b1700 1 -- 172.21.15.146:0/2930086650 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44e80f62a0 msgr2=0x7f44e80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:15.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:15.238+0000 7f44ed9b1700 1 --2- 172.21.15.146:0/2930086650 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44e80f62a0 0x7f44e80f66c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f44dc009fb0 tx=0x7f44dc009b30).stop 2024-01-21T20:06:15.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:15.239+0000 7f44ed9b1700 1 -- 172.21.15.146:0/2930086650 shutdown_connections 2024-01-21T20:06:15.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:15.239+0000 7f44ed9b1700 1 --2- 172.21.15.146:0/2930086650 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44e80f62a0 0x7f44e80f66c0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:15.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:15.239+0000 7f44ed9b1700 1 -- 172.21.15.146:0/2930086650 >> 172.21.15.146:0/2930086650 conn(0x7f44e80f1aa0 msgr2=0x7f44e80f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:15.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:15.239+0000 7f44ed9b1700 1 -- 172.21.15.146:0/2930086650 shutdown_connections 2024-01-21T20:06:15.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:15.239+0000 7f44ed9b1700 1 -- 172.21.15.146:0/2930086650 wait complete. 2024-01-21T20:06:15.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:15.239+0000 7f44ed9b1700 1 Processor -- start 2024-01-21T20:06:15.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:15.239+0000 7f44ed9b1700 1 -- start start 2024-01-21T20:06:15.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:15.239+0000 7f44ed9b1700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44e80f62a0 0x7f44e81064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:15.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:15.239+0000 7f44ed9b1700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e81069a0 con 0x7f44e80f62a0 2024-01-21T20:06:15.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:15.240+0000 7f44ec9af700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44e80f62a0 0x7f44e81064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:15.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:15.240+0000 7f44ec9af700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44e80f62a0 0x7f44e81064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:60120/0 (socket says 172.21.15.146:60120) 2024-01-21T20:06:15.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:15.240+0000 7f44ec9af700 1 -- 172.21.15.146:0/3552240853 learned_addr learned my addr 172.21.15.146:0/3552240853 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:06:15.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:15.240+0000 7f44ec9af700 1 -- 172.21.15.146:0/3552240853 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44dc005040 con 0x7f44e80f62a0 2024-01-21T20:06:15.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:15.240+0000 7f44ec9af700 1 --2- 172.21.15.146:0/3552240853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44e80f62a0 0x7f44e81064b0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f44dc013510 tx=0x7f44dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:15.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:15.240+0000 7f44e57fa700 1 -- 172.21.15.146:0/3552240853 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44dc009320 con 0x7f44e80f62a0 2024-01-21T20:06:15.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:15.240+0000 7f44ed9b1700 1 -- 172.21.15.146:0/3552240853 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44e8106ba0 con 0x7f44e80f62a0 2024-01-21T20:06:15.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:15.240+0000 7f44e57fa700 1 -- 172.21.15.146:0/3552240853 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f44dc013f00 con 0x7f44e80f62a0 2024-01-21T20:06:15.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:15.240+0000 7f44e57fa700 1 -- 172.21.15.146:0/3552240853 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44dc014dc0 con 0x7f44e80f62a0 2024-01-21T20:06:15.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:15.240+0000 7f44ed9b1700 1 -- 172.21.15.146:0/3552240853 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44e8107030 con 0x7f44e80f62a0 2024-01-21T20:06:15.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:15.241+0000 7f44e57fa700 1 -- 172.21.15.146:0/3552240853 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f44dc0095f0 con 0x7f44e80f62a0 2024-01-21T20:06:15.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:15.241+0000 7f44e57fa700 1 --2- 172.21.15.146:0/3552240853 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f44d802ff60 0x7f44d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:15.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:15.241+0000 7f44ed9b1700 1 -- 172.21.15.146:0/3552240853 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44e8045ad0 con 0x7f44e80f62a0 2024-01-21T20:06:15.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:15.241+0000 7f44e57fa700 1 -- 172.21.15.146:0/3552240853 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f44dc041100 con 0x7f44e80f62a0 2024-01-21T20:06:15.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:15.242+0000 7f44e7fff700 1 --2- 172.21.15.146:0/3552240853 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f44d802ff60 0x7f44d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:15.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:15.242+0000 7f44e7fff700 1 --2- 172.21.15.146:0/3552240853 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f44d802ff60 0x7f44d8032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f44d40096a0 tx=0x7f44d4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:15.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:15.246+0000 7f44e57fa700 1 -- 172.21.15.146:0/3552240853 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f44dc014420 con 0x7f44e80f62a0 2024-01-21T20:06:15.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:15.452+0000 7f44ed9b1700 1 -- 172.21.15.146:0/3552240853 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f44e8107ee0 con 0x7f44e80f62a0 2024-01-21T20:06:15.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:15.453+0000 7f44e57fa700 1 -- 172.21.15.146:0/3552240853 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f44dc02b020 con 0x7f44e80f62a0 2024-01-21T20:06:15.444 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:06:15.444 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:06:15.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:15.455+0000 7f44ed9b1700 1 -- 172.21.15.146:0/3552240853 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f44d802ff60 msgr2=0x7f44d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:15.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:15.455+0000 7f44ed9b1700 1 --2- 172.21.15.146:0/3552240853 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f44d802ff60 0x7f44d8032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f44d40096a0 tx=0x7f44d4006b40).stop 2024-01-21T20:06:15.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:15.455+0000 7f44ed9b1700 1 -- 172.21.15.146:0/3552240853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44e80f62a0 msgr2=0x7f44e81064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:15.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:15.455+0000 7f44ed9b1700 1 --2- 172.21.15.146:0/3552240853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44e80f62a0 0x7f44e81064b0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f44dc013510 tx=0x7f44dc00b040).stop 2024-01-21T20:06:15.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:15.455+0000 7f44ed9b1700 1 -- 172.21.15.146:0/3552240853 shutdown_connections 2024-01-21T20:06:15.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:15.455+0000 7f44ed9b1700 1 --2- 172.21.15.146:0/3552240853 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f44d802ff60 0x7f44d8032420 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:15.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:15.455+0000 7f44ed9b1700 1 --2- 172.21.15.146:0/3552240853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44e80f62a0 0x7f44e81064b0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:15.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:15.455+0000 7f44ed9b1700 1 -- 172.21.15.146:0/3552240853 >> 172.21.15.146:0/3552240853 conn(0x7f44e80f1aa0 msgr2=0x7f44e80f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:15.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:15.455+0000 7f44ed9b1700 1 -- 172.21.15.146:0/3552240853 shutdown_connections 2024-01-21T20:06:15.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:15.455+0000 7f44ed9b1700 1 -- 172.21.15.146:0/3552240853 wait complete. 2024-01-21T20:06:15.447 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:06:16.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:15 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:06:15.454556+0000 mon.smithi053 (mon.0) 279 : audit [DBG] from='client.? 172.21.15.146:0/3552240853' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:06:16.873 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:06:16.873 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:06:17.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:06:15.193045+0000 mgr.smithi053.fgsmol (mgr.14164) 219 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:18.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:18.124+0000 7f93d9471700 1 -- 172.21.15.146:0/2720882703 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93d40f51a0 msgr2=0x7f93d40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:18.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:18.124+0000 7f93d9471700 1 --2- 172.21.15.146:0/2720882703 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93d40f51a0 0x7f93d40f55c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f93c4009fb0 tx=0x7f93c4009b30).stop 2024-01-21T20:06:18.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:18.125+0000 7f93d9471700 1 -- 172.21.15.146:0/2720882703 shutdown_connections 2024-01-21T20:06:18.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:18.125+0000 7f93d9471700 1 --2- 172.21.15.146:0/2720882703 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93d40f51a0 0x7f93d40f55c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:18.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:18.125+0000 7f93d9471700 1 -- 172.21.15.146:0/2720882703 >> 172.21.15.146:0/2720882703 conn(0x7f93d40f09a0 msgr2=0x7f93d40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:18.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:18.125+0000 7f93d9471700 1 -- 172.21.15.146:0/2720882703 shutdown_connections 2024-01-21T20:06:18.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:18.125+0000 7f93d9471700 1 -- 172.21.15.146:0/2720882703 wait complete. 2024-01-21T20:06:18.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:18.125+0000 7f93d9471700 1 Processor -- start 2024-01-21T20:06:18.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:18.125+0000 7f93d9471700 1 -- start start 2024-01-21T20:06:18.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:18.125+0000 7f93d9471700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93d40f51a0 0x7f93d410b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:18.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:18.125+0000 7f93d9471700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d4106fc0 con 0x7f93d40f51a0 2024-01-21T20:06:18.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:18.126+0000 7f93d3fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93d40f51a0 0x7f93d410b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:18.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:18.126+0000 7f93d3fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93d40f51a0 0x7f93d410b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:51702/0 (socket says 172.21.15.146:51702) 2024-01-21T20:06:18.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:18.126+0000 7f93d3fff700 1 -- 172.21.15.146:0/3009835777 learned_addr learned my addr 172.21.15.146:0/3009835777 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:06:18.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:18.126+0000 7f93d3fff700 1 -- 172.21.15.146:0/3009835777 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93c4005040 con 0x7f93d40f51a0 2024-01-21T20:06:18.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:18.126+0000 7f93d3fff700 1 --2- 172.21.15.146:0/3009835777 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93d40f51a0 0x7f93d410b5d0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f93c4013510 tx=0x7f93c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:18.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:18.126+0000 7f93d0ff9700 1 -- 172.21.15.146:0/3009835777 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f93c4009320 con 0x7f93d40f51a0 2024-01-21T20:06:18.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:18.126+0000 7f93d0ff9700 1 -- 172.21.15.146:0/3009835777 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f93c4013f00 con 0x7f93d40f51a0 2024-01-21T20:06:18.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:18.126+0000 7f93d9471700 1 -- 172.21.15.146:0/3009835777 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93d4060220 con 0x7f93d40f51a0 2024-01-21T20:06:18.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:18.126+0000 7f93d0ff9700 1 -- 172.21.15.146:0/3009835777 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f93c4014dc0 con 0x7f93d40f51a0 2024-01-21T20:06:18.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:18.126+0000 7f93d9471700 1 -- 172.21.15.146:0/3009835777 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93d410bac0 con 0x7f93d40f51a0 2024-01-21T20:06:18.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:18.127+0000 7f93d0ff9700 1 -- 172.21.15.146:0/3009835777 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f93c401d3d0 con 0x7f93d40f51a0 2024-01-21T20:06:18.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:18.127+0000 7f93d9471700 1 -- 172.21.15.146:0/3009835777 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93d4045ad0 con 0x7f93d40f51a0 2024-01-21T20:06:18.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:18.127+0000 7f93d0ff9700 1 --2- 172.21.15.146:0/3009835777 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f93bc02ff60 0x7f93bc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:18.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:18.127+0000 7f93d0ff9700 1 -- 172.21.15.146:0/3009835777 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f93c40413c0 con 0x7f93d40f51a0 2024-01-21T20:06:18.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:18.128+0000 7f93d37fe700 1 --2- 172.21.15.146:0/3009835777 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f93bc02ff60 0x7f93bc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:18.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:18.128+0000 7f93d37fe700 1 --2- 172.21.15.146:0/3009835777 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f93bc02ff60 0x7f93bc032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f93c80096a0 tx=0x7f93c8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:18.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:18.132+0000 7f93d0ff9700 1 -- 172.21.15.146:0/3009835777 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f93c4012510 con 0x7f93d40f51a0 2024-01-21T20:06:18.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:18.336+0000 7f93d9471700 1 -- 172.21.15.146:0/3009835777 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f93d410c280 con 0x7f93d40f51a0 2024-01-21T20:06:18.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:18.337+0000 7f93d0ff9700 1 -- 172.21.15.146:0/3009835777 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f93c4012330 con 0x7f93d40f51a0 2024-01-21T20:06:18.328 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:06:18.328 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:06:18.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:18.339+0000 7f93d9471700 1 -- 172.21.15.146:0/3009835777 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f93bc02ff60 msgr2=0x7f93bc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:18.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:18.339+0000 7f93d9471700 1 --2- 172.21.15.146:0/3009835777 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f93bc02ff60 0x7f93bc032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f93c80096a0 tx=0x7f93c8006b40).stop 2024-01-21T20:06:18.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:18.339+0000 7f93d9471700 1 -- 172.21.15.146:0/3009835777 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93d40f51a0 msgr2=0x7f93d410b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:18.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:18.339+0000 7f93d9471700 1 --2- 172.21.15.146:0/3009835777 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93d40f51a0 0x7f93d410b5d0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f93c4013510 tx=0x7f93c400b040).stop 2024-01-21T20:06:18.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:18.339+0000 7f93d9471700 1 -- 172.21.15.146:0/3009835777 shutdown_connections 2024-01-21T20:06:18.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:18.339+0000 7f93d9471700 1 --2- 172.21.15.146:0/3009835777 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f93bc02ff60 0x7f93bc032420 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:18.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:18.339+0000 7f93d9471700 1 --2- 172.21.15.146:0/3009835777 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93d40f51a0 0x7f93d410b5d0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:18.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:18.339+0000 7f93d9471700 1 -- 172.21.15.146:0/3009835777 >> 172.21.15.146:0/3009835777 conn(0x7f93d40f09a0 msgr2=0x7f93d40f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:18.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:18.339+0000 7f93d9471700 1 -- 172.21.15.146:0/3009835777 shutdown_connections 2024-01-21T20:06:18.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:18.339+0000 7f93d9471700 1 -- 172.21.15.146:0/3009835777 wait complete. 2024-01-21T20:06:18.330 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:06:19.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:18 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:06:17.193392+0000 mgr.smithi053.fgsmol (mgr.14164) 220 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:19.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:18 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:06:18.338414+0000 mon.smithi053 (mon.0) 280 : audit [DBG] from='client.? 172.21.15.146:0/3009835777' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:06:19.783 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:06:19.784 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:06:20.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:21.005+0000 7f66a7380700 1 -- 172.21.15.146:0/1816242140 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66a00f5410 msgr2=0x7f66a00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:20.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:21.005+0000 7f66a7380700 1 --2- 172.21.15.146:0/1816242140 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66a00f5410 0x7f66a00f5830 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f6690009fb0 tx=0x7f6690009b30).stop 2024-01-21T20:06:20.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:21.005+0000 7f66a7380700 1 -- 172.21.15.146:0/1816242140 shutdown_connections 2024-01-21T20:06:20.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:21.005+0000 7f66a7380700 1 --2- 172.21.15.146:0/1816242140 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66a00f5410 0x7f66a00f5830 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:20.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:21.005+0000 7f66a7380700 1 -- 172.21.15.146:0/1816242140 >> 172.21.15.146:0/1816242140 conn(0x7f66a00f0b70 msgr2=0x7f66a00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:20.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:21.005+0000 7f66a7380700 1 -- 172.21.15.146:0/1816242140 shutdown_connections 2024-01-21T20:06:20.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:21.005+0000 7f66a7380700 1 -- 172.21.15.146:0/1816242140 wait complete. 2024-01-21T20:06:20.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:21.006+0000 7f66a7380700 1 Processor -- start 2024-01-21T20:06:20.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:21.006+0000 7f66a7380700 1 -- start start 2024-01-21T20:06:21.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:21.006+0000 7f66a7380700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66a00f5410 0x7f66a0105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:21.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:21.006+0000 7f66a7380700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a0105b20 con 0x7f66a00f5410 2024-01-21T20:06:21.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:21.006+0000 7f66a511c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66a00f5410 0x7f66a0105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:21.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:21.007+0000 7f66a511c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66a00f5410 0x7f66a0105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:51726/0 (socket says 172.21.15.146:51726) 2024-01-21T20:06:21.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:21.007+0000 7f66a511c700 1 -- 172.21.15.146:0/1815192159 learned_addr learned my addr 172.21.15.146:0/1815192159 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:06:21.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:21.007+0000 7f66a511c700 1 -- 172.21.15.146:0/1815192159 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6690005040 con 0x7f66a00f5410 2024-01-21T20:06:21.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:21.007+0000 7f66a511c700 1 --2- 172.21.15.146:0/1815192159 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66a00f5410 0x7f66a0105630 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f6690013510 tx=0x7f669000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:21.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:21.007+0000 7f6695ffb700 1 -- 172.21.15.146:0/1815192159 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6690009320 con 0x7f66a00f5410 2024-01-21T20:06:21.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:21.007+0000 7f66a7380700 1 -- 172.21.15.146:0/1815192159 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66a0105d20 con 0x7f66a00f5410 2024-01-21T20:06:21.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:21.007+0000 7f6695ffb700 1 -- 172.21.15.146:0/1815192159 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6690013f00 con 0x7f66a00f5410 2024-01-21T20:06:21.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:21.007+0000 7f6695ffb700 1 -- 172.21.15.146:0/1815192159 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6690014d90 con 0x7f66a00f5410 2024-01-21T20:06:21.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:21.007+0000 7f66a7380700 1 -- 172.21.15.146:0/1815192159 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66a01061b0 con 0x7f66a00f5410 2024-01-21T20:06:21.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:21.008+0000 7f6695ffb700 1 -- 172.21.15.146:0/1815192159 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6690014420 con 0x7f66a00f5410 2024-01-21T20:06:21.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:21.008+0000 7f66a7380700 1 -- 172.21.15.146:0/1815192159 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66a0045ad0 con 0x7f66a00f5410 2024-01-21T20:06:21.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:21.008+0000 7f6695ffb700 1 --2- 172.21.15.146:0/1815192159 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f668c02ff60 0x7f668c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:21.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:21.008+0000 7f6695ffb700 1 -- 172.21.15.146:0/1815192159 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f66900410a0 con 0x7f66a00f5410 2024-01-21T20:06:21.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:21.009+0000 7f66a491b700 1 --2- 172.21.15.146:0/1815192159 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f668c02ff60 0x7f668c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:21.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:21.009+0000 7f66a491b700 1 --2- 172.21.15.146:0/1815192159 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f668c02ff60 0x7f668c032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f669c0096a0 tx=0x7f669c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:21.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:21.013+0000 7f6695ffb700 1 -- 172.21.15.146:0/1815192159 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f66900095f0 con 0x7f66a00f5410 2024-01-21T20:06:21.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:20 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:06:19.193723+0000 mgr.smithi053.fgsmol (mgr.14164) 221 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:21.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:21.217+0000 7f66a7380700 1 -- 172.21.15.146:0/1815192159 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f66a00fae60 con 0x7f66a00f5410 2024-01-21T20:06:21.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:21.218+0000 7f6695ffb700 1 -- 172.21.15.146:0/1815192159 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f669001db70 con 0x7f66a00f5410 2024-01-21T20:06:21.209 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:06:21.209 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:06:21.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:21.219+0000 7f66a7380700 1 -- 172.21.15.146:0/1815192159 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f668c02ff60 msgr2=0x7f668c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:21.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:21.219+0000 7f66a7380700 1 --2- 172.21.15.146:0/1815192159 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f668c02ff60 0x7f668c032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f669c0096a0 tx=0x7f669c006b40).stop 2024-01-21T20:06:21.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:21.219+0000 7f66a7380700 1 -- 172.21.15.146:0/1815192159 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66a00f5410 msgr2=0x7f66a0105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:21.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:21.219+0000 7f66a7380700 1 --2- 172.21.15.146:0/1815192159 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66a00f5410 0x7f66a0105630 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f6690013510 tx=0x7f669000b040).stop 2024-01-21T20:06:21.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:21.219+0000 7f66a7380700 1 -- 172.21.15.146:0/1815192159 shutdown_connections 2024-01-21T20:06:21.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:21.219+0000 7f66a7380700 1 --2- 172.21.15.146:0/1815192159 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f668c02ff60 0x7f668c032420 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:21.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:21.219+0000 7f66a7380700 1 --2- 172.21.15.146:0/1815192159 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66a00f5410 0x7f66a0105630 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:21.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:21.219+0000 7f66a7380700 1 -- 172.21.15.146:0/1815192159 >> 172.21.15.146:0/1815192159 conn(0x7f66a00f0b70 msgr2=0x7f66a00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:21.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:21.220+0000 7f66a7380700 1 -- 172.21.15.146:0/1815192159 shutdown_connections 2024-01-21T20:06:21.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:21.220+0000 7f66a7380700 1 -- 172.21.15.146:0/1815192159 wait complete. 2024-01-21T20:06:21.211 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:06:22.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:21 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:06:21.219199+0000 mon.smithi053 (mon.0) 281 : audit [DBG] from='client.? 172.21.15.146:0/1815192159' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:06:22.664 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:06:22.664 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:06:23.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:06:21.194027+0000 mgr.smithi053.fgsmol (mgr.14164) 222 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:23.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:23.910+0000 7f0d1b321700 1 -- 172.21.15.146:0/1580525920 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d140f5410 msgr2=0x7f0d140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:23.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:23.910+0000 7f0d1b321700 1 --2- 172.21.15.146:0/1580525920 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d140f5410 0x7f0d140f5830 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f0d04009fb0 tx=0x7f0d04009b30).stop 2024-01-21T20:06:23.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:23.910+0000 7f0d1b321700 1 -- 172.21.15.146:0/1580525920 shutdown_connections 2024-01-21T20:06:23.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:23.910+0000 7f0d1b321700 1 --2- 172.21.15.146:0/1580525920 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d140f5410 0x7f0d140f5830 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:23.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:23.910+0000 7f0d1b321700 1 -- 172.21.15.146:0/1580525920 >> 172.21.15.146:0/1580525920 conn(0x7f0d140f0b70 msgr2=0x7f0d140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:23.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:23.910+0000 7f0d1b321700 1 -- 172.21.15.146:0/1580525920 shutdown_connections 2024-01-21T20:06:23.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:23.910+0000 7f0d1b321700 1 -- 172.21.15.146:0/1580525920 wait complete. 2024-01-21T20:06:23.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:23.911+0000 7f0d1b321700 1 Processor -- start 2024-01-21T20:06:23.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:23.911+0000 7f0d1b321700 1 -- start start 2024-01-21T20:06:23.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:23.911+0000 7f0d1b321700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d140f5410 0x7f0d14105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:23.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:23.911+0000 7f0d1b321700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d14105b20 con 0x7f0d140f5410 2024-01-21T20:06:23.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:23.911+0000 7f0d190bd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d140f5410 0x7f0d14105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:23.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:23.911+0000 7f0d190bd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d140f5410 0x7f0d14105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:51748/0 (socket says 172.21.15.146:51748) 2024-01-21T20:06:23.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:23.911+0000 7f0d190bd700 1 -- 172.21.15.146:0/2569103676 learned_addr learned my addr 172.21.15.146:0/2569103676 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:06:23.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:23.911+0000 7f0d190bd700 1 -- 172.21.15.146:0/2569103676 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d04005040 con 0x7f0d140f5410 2024-01-21T20:06:23.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:23.911+0000 7f0d190bd700 1 --2- 172.21.15.146:0/2569103676 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d140f5410 0x7f0d14105630 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f0d04013510 tx=0x7f0d0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:23.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:23.912+0000 7f0d09ffb700 1 -- 172.21.15.146:0/2569103676 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d04009320 con 0x7f0d140f5410 2024-01-21T20:06:23.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:23.912+0000 7f0d1b321700 1 -- 172.21.15.146:0/2569103676 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d14105d20 con 0x7f0d140f5410 2024-01-21T20:06:23.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:23.912+0000 7f0d09ffb700 1 -- 172.21.15.146:0/2569103676 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d04013f00 con 0x7f0d140f5410 2024-01-21T20:06:23.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:23.912+0000 7f0d09ffb700 1 -- 172.21.15.146:0/2569103676 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d04014d90 con 0x7f0d140f5410 2024-01-21T20:06:23.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:23.912+0000 7f0d1b321700 1 -- 172.21.15.146:0/2569103676 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d141061b0 con 0x7f0d140f5410 2024-01-21T20:06:23.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:23.913+0000 7f0d09ffb700 1 -- 172.21.15.146:0/2569103676 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0d040095f0 con 0x7f0d140f5410 2024-01-21T20:06:23.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:23.913+0000 7f0d1b321700 1 -- 172.21.15.146:0/2569103676 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d14045ad0 con 0x7f0d140f5410 2024-01-21T20:06:23.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:23.913+0000 7f0d09ffb700 1 --2- 172.21.15.146:0/2569103676 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0d0002ff60 0x7f0d00032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:23.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:23.913+0000 7f0d09ffb700 1 -- 172.21.15.146:0/2569103676 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0d04041000 con 0x7f0d140f5410 2024-01-21T20:06:23.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:23.913+0000 7f0d188bc700 1 --2- 172.21.15.146:0/2569103676 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0d0002ff60 0x7f0d00032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:23.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:23.914+0000 7f0d188bc700 1 --2- 172.21.15.146:0/2569103676 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0d0002ff60 0x7f0d00032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f0d100096a0 tx=0x7f0d10006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:23.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:23.918+0000 7f0d09ffb700 1 -- 172.21.15.146:0/2569103676 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0d04014420 con 0x7f0d140f5410 2024-01-21T20:06:24.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:24.122+0000 7f0d1b321700 1 -- 172.21.15.146:0/2569103676 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0d140fae60 con 0x7f0d140f5410 2024-01-21T20:06:24.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:24.122+0000 7f0d09ffb700 1 -- 172.21.15.146:0/2569103676 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0d0401d7f0 con 0x7f0d140f5410 2024-01-21T20:06:24.113 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:06:24.113 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:06:24.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:24.124+0000 7f0d1b321700 1 -- 172.21.15.146:0/2569103676 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0d0002ff60 msgr2=0x7f0d00032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:24.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:24.124+0000 7f0d1b321700 1 --2- 172.21.15.146:0/2569103676 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0d0002ff60 0x7f0d00032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f0d100096a0 tx=0x7f0d10006b40).stop 2024-01-21T20:06:24.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:24.124+0000 7f0d1b321700 1 -- 172.21.15.146:0/2569103676 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d140f5410 msgr2=0x7f0d14105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:24.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:24.124+0000 7f0d1b321700 1 --2- 172.21.15.146:0/2569103676 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d140f5410 0x7f0d14105630 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f0d04013510 tx=0x7f0d0400b040).stop 2024-01-21T20:06:24.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:24.124+0000 7f0d1b321700 1 -- 172.21.15.146:0/2569103676 shutdown_connections 2024-01-21T20:06:24.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:24.124+0000 7f0d1b321700 1 --2- 172.21.15.146:0/2569103676 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0d0002ff60 0x7f0d00032420 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:24.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:24.124+0000 7f0d1b321700 1 --2- 172.21.15.146:0/2569103676 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d140f5410 0x7f0d14105630 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:24.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:24.124+0000 7f0d1b321700 1 -- 172.21.15.146:0/2569103676 >> 172.21.15.146:0/2569103676 conn(0x7f0d140f0b70 msgr2=0x7f0d140f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:24.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:24.124+0000 7f0d1b321700 1 -- 172.21.15.146:0/2569103676 shutdown_connections 2024-01-21T20:06:24.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:24.124+0000 7f0d1b321700 1 -- 172.21.15.146:0/2569103676 wait complete. 2024-01-21T20:06:24.116 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:06:25.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:24 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:06:23.194317+0000 mgr.smithi053.fgsmol (mgr.14164) 223 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:25.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:24 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:06:24.123854+0000 mon.smithi053 (mon.0) 282 : audit [DBG] from='client.? 172.21.15.146:0/2569103676' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:06:25.517 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:06:25.517 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:06:26.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:26.731+0000 7f59fe9bd700 1 -- 172.21.15.146:0/3205707484 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59f80f5410 msgr2=0x7f59f80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:26.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:26.731+0000 7f59fe9bd700 1 --2- 172.21.15.146:0/3205707484 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59f80f5410 0x7f59f80f5830 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f59e8009fb0 tx=0x7f59e8009b30).stop 2024-01-21T20:06:26.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:26.731+0000 7f59fe9bd700 1 -- 172.21.15.146:0/3205707484 shutdown_connections 2024-01-21T20:06:26.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:26.731+0000 7f59fe9bd700 1 --2- 172.21.15.146:0/3205707484 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59f80f5410 0x7f59f80f5830 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:26.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:26.731+0000 7f59fe9bd700 1 -- 172.21.15.146:0/3205707484 >> 172.21.15.146:0/3205707484 conn(0x7f59f80f0b70 msgr2=0x7f59f80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:26.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:26.731+0000 7f59fe9bd700 1 -- 172.21.15.146:0/3205707484 shutdown_connections 2024-01-21T20:06:26.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:26.731+0000 7f59fe9bd700 1 -- 172.21.15.146:0/3205707484 wait complete. 2024-01-21T20:06:26.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:26.731+0000 7f59fe9bd700 1 Processor -- start 2024-01-21T20:06:26.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:26.732+0000 7f59fe9bd700 1 -- start start 2024-01-21T20:06:26.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:26.732+0000 7f59fe9bd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59f80f5410 0x7f59f8109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:26.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:26.732+0000 7f59fe9bd700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f8060240 con 0x7f59f80f5410 2024-01-21T20:06:26.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:26.732+0000 7f59f7fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59f80f5410 0x7f59f8109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:26.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:26.732+0000 7f59f7fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59f80f5410 0x7f59f8109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:51758/0 (socket says 172.21.15.146:51758) 2024-01-21T20:06:26.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:26.732+0000 7f59f7fff700 1 -- 172.21.15.146:0/3864694647 learned_addr learned my addr 172.21.15.146:0/3864694647 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:06:26.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:26.733+0000 7f59f7fff700 1 -- 172.21.15.146:0/3864694647 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e8005040 con 0x7f59f80f5410 2024-01-21T20:06:26.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:26.733+0000 7f59f7fff700 1 --2- 172.21.15.146:0/3864694647 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59f80f5410 0x7f59f8109430 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f59e8013510 tx=0x7f59e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:26.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:26.733+0000 7f59f4ff9700 1 -- 172.21.15.146:0/3864694647 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59e8009320 con 0x7f59f80f5410 2024-01-21T20:06:26.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:26.733+0000 7f59fe9bd700 1 -- 172.21.15.146:0/3864694647 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59f8060440 con 0x7f59f80f5410 2024-01-21T20:06:26.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:26.733+0000 7f59f4ff9700 1 -- 172.21.15.146:0/3864694647 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f59e8013f00 con 0x7f59f80f5410 2024-01-21T20:06:26.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:26.733+0000 7f59f4ff9700 1 -- 172.21.15.146:0/3864694647 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59e8014d90 con 0x7f59f80f5410 2024-01-21T20:06:26.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:26.733+0000 7f59fe9bd700 1 -- 172.21.15.146:0/3864694647 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59f8109ae0 con 0x7f59f80f5410 2024-01-21T20:06:26.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:26.734+0000 7f59f4ff9700 1 -- 172.21.15.146:0/3864694647 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f59e8014420 con 0x7f59f80f5410 2024-01-21T20:06:26.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:26.734+0000 7f59fe9bd700 1 -- 172.21.15.146:0/3864694647 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59f8045ad0 con 0x7f59f80f5410 2024-01-21T20:06:26.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:26.734+0000 7f59f4ff9700 1 --2- 172.21.15.146:0/3864694647 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f59e002ff60 0x7f59e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:26.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:26.734+0000 7f59f4ff9700 1 -- 172.21.15.146:0/3864694647 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f59e80410c0 con 0x7f59f80f5410 2024-01-21T20:06:26.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:26.734+0000 7f59f77fe700 1 --2- 172.21.15.146:0/3864694647 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f59e002ff60 0x7f59e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:26.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:26.735+0000 7f59f77fe700 1 --2- 172.21.15.146:0/3864694647 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f59e002ff60 0x7f59e0032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f59ec0096a0 tx=0x7f59ec006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:26.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:26.739+0000 7f59f4ff9700 1 -- 172.21.15.146:0/3864694647 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f59e80095f0 con 0x7f59f80f5410 2024-01-21T20:06:26.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:26.944+0000 7f59fe9bd700 1 -- 172.21.15.146:0/3864694647 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f59f80fae60 con 0x7f59f80f5410 2024-01-21T20:06:26.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:26.945+0000 7f59f4ff9700 1 -- 172.21.15.146:0/3864694647 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f59e801db70 con 0x7f59f80f5410 2024-01-21T20:06:26.936 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:06:26.936 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:06:26.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:26.946+0000 7f59fe9bd700 1 -- 172.21.15.146:0/3864694647 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f59e002ff60 msgr2=0x7f59e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:26.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:26.946+0000 7f59fe9bd700 1 --2- 172.21.15.146:0/3864694647 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f59e002ff60 0x7f59e0032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f59ec0096a0 tx=0x7f59ec006b40).stop 2024-01-21T20:06:26.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:26.946+0000 7f59fe9bd700 1 -- 172.21.15.146:0/3864694647 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59f80f5410 msgr2=0x7f59f8109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:26.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:26.946+0000 7f59fe9bd700 1 --2- 172.21.15.146:0/3864694647 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59f80f5410 0x7f59f8109430 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f59e8013510 tx=0x7f59e800b040).stop 2024-01-21T20:06:26.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:26.946+0000 7f59fe9bd700 1 -- 172.21.15.146:0/3864694647 shutdown_connections 2024-01-21T20:06:26.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:26.946+0000 7f59fe9bd700 1 --2- 172.21.15.146:0/3864694647 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f59e002ff60 0x7f59e0032420 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:26.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:26.946+0000 7f59fe9bd700 1 --2- 172.21.15.146:0/3864694647 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59f80f5410 0x7f59f8109430 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:26.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:26.946+0000 7f59fe9bd700 1 -- 172.21.15.146:0/3864694647 >> 172.21.15.146:0/3864694647 conn(0x7f59f80f0b70 msgr2=0x7f59f80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:26.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:26.947+0000 7f59fe9bd700 1 -- 172.21.15.146:0/3864694647 shutdown_connections 2024-01-21T20:06:26.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:26.947+0000 7f59fe9bd700 1 -- 172.21.15.146:0/3864694647 wait complete. 2024-01-21T20:06:26.938 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:06:27.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:26 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:06:25.194572+0000 mgr.smithi053.fgsmol (mgr.14164) 224 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:28.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:27 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:06:26.946424+0000 mon.smithi053 (mon.0) 283 : audit [DBG] from='client.? 172.21.15.146:0/3864694647' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:06:28.350 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:06:28.350 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:06:29.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:28 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:06:27.194828+0000 mgr.smithi053.fgsmol (mgr.14164) 225 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:29.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:29.651+0000 7f55c9ed3700 1 -- 172.21.15.146:0/2054014112 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55c40f5410 msgr2=0x7f55c40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:29.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:29.651+0000 7f55c9ed3700 1 --2- 172.21.15.146:0/2054014112 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55c40f5410 0x7f55c40f5830 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f55b4009fb0 tx=0x7f55b4009b30).stop 2024-01-21T20:06:29.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:29.651+0000 7f55c9ed3700 1 -- 172.21.15.146:0/2054014112 shutdown_connections 2024-01-21T20:06:29.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:29.651+0000 7f55c9ed3700 1 --2- 172.21.15.146:0/2054014112 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55c40f5410 0x7f55c40f5830 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:29.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:29.651+0000 7f55c9ed3700 1 -- 172.21.15.146:0/2054014112 >> 172.21.15.146:0/2054014112 conn(0x7f55c40f0b70 msgr2=0x7f55c40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:29.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:29.651+0000 7f55c9ed3700 1 -- 172.21.15.146:0/2054014112 shutdown_connections 2024-01-21T20:06:29.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:29.651+0000 7f55c9ed3700 1 -- 172.21.15.146:0/2054014112 wait complete. 2024-01-21T20:06:29.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:29.651+0000 7f55c9ed3700 1 Processor -- start 2024-01-21T20:06:29.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:29.651+0000 7f55c9ed3700 1 -- start start 2024-01-21T20:06:29.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:29.652+0000 7f55c9ed3700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55c40f5410 0x7f55c4109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:29.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:29.652+0000 7f55c9ed3700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55c4060220 con 0x7f55c40f5410 2024-01-21T20:06:29.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:29.652+0000 7f55c37fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55c40f5410 0x7f55c4109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:29.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:29.652+0000 7f55c37fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55c40f5410 0x7f55c4109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:56396/0 (socket says 172.21.15.146:56396) 2024-01-21T20:06:29.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:29.652+0000 7f55c37fe700 1 -- 172.21.15.146:0/2818423163 learned_addr learned my addr 172.21.15.146:0/2818423163 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:06:29.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:29.652+0000 7f55c37fe700 1 -- 172.21.15.146:0/2818423163 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b4005040 con 0x7f55c40f5410 2024-01-21T20:06:29.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:29.652+0000 7f55c37fe700 1 --2- 172.21.15.146:0/2818423163 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55c40f5410 0x7f55c4109450 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f55b4014000 tx=0x7f55b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:29.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:29.653+0000 7f55abfff700 1 -- 172.21.15.146:0/2818423163 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55b40095f0 con 0x7f55c40f5410 2024-01-21T20:06:29.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:29.653+0000 7f55c9ed3700 1 -- 172.21.15.146:0/2818423163 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55c4060420 con 0x7f55c40f5410 2024-01-21T20:06:29.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:29.653+0000 7f55abfff700 1 -- 172.21.15.146:0/2818423163 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f55b4007e70 con 0x7f55c40f5410 2024-01-21T20:06:29.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:29.653+0000 7f55abfff700 1 -- 172.21.15.146:0/2818423163 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55b4014ea0 con 0x7f55c40f5410 2024-01-21T20:06:29.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:29.653+0000 7f55c9ed3700 1 -- 172.21.15.146:0/2818423163 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55c4109b00 con 0x7f55c40f5410 2024-01-21T20:06:29.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:29.653+0000 7f55abfff700 1 -- 172.21.15.146:0/2818423163 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f55b4025070 con 0x7f55c40f5410 2024-01-21T20:06:29.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:29.653+0000 7f55c9ed3700 1 -- 172.21.15.146:0/2818423163 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55c4059100 con 0x7f55c40f5410 2024-01-21T20:06:29.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:29.654+0000 7f55abfff700 1 --2- 172.21.15.146:0/2818423163 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f55ac02ff60 0x7f55ac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:29.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:29.654+0000 7f55abfff700 1 -- 172.21.15.146:0/2818423163 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f55b4040970 con 0x7f55c40f5410 2024-01-21T20:06:29.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:29.654+0000 7f55c2ffd700 1 --2- 172.21.15.146:0/2818423163 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f55ac02ff60 0x7f55ac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:29.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:29.654+0000 7f55c2ffd700 1 --2- 172.21.15.146:0/2818423163 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f55ac02ff60 0x7f55ac032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f55b80096a0 tx=0x7f55b8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:29.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:29.658+0000 7f55abfff700 1 -- 172.21.15.146:0/2818423163 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f55b400f4e0 con 0x7f55c40f5410 2024-01-21T20:06:29.853 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:29.862+0000 7f55c9ed3700 1 -- 172.21.15.146:0/2818423163 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f55c4045ad0 con 0x7f55c40f5410 2024-01-21T20:06:29.854 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:29.863+0000 7f55abfff700 1 -- 172.21.15.146:0/2818423163 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f55b400f300 con 0x7f55c40f5410 2024-01-21T20:06:29.854 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:06:29.854 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:06:29.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:29.865+0000 7f55c9ed3700 1 -- 172.21.15.146:0/2818423163 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f55ac02ff60 msgr2=0x7f55ac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:29.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:29.865+0000 7f55c9ed3700 1 --2- 172.21.15.146:0/2818423163 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f55ac02ff60 0x7f55ac032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f55b80096a0 tx=0x7f55b8006b40).stop 2024-01-21T20:06:29.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:29.865+0000 7f55c9ed3700 1 -- 172.21.15.146:0/2818423163 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55c40f5410 msgr2=0x7f55c4109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:29.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:29.865+0000 7f55c9ed3700 1 --2- 172.21.15.146:0/2818423163 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55c40f5410 0x7f55c4109450 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f55b4014000 tx=0x7f55b400b040).stop 2024-01-21T20:06:29.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:29.865+0000 7f55c9ed3700 1 -- 172.21.15.146:0/2818423163 shutdown_connections 2024-01-21T20:06:29.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:29.865+0000 7f55c9ed3700 1 --2- 172.21.15.146:0/2818423163 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f55ac02ff60 0x7f55ac032420 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:29.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:29.865+0000 7f55c9ed3700 1 --2- 172.21.15.146:0/2818423163 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55c40f5410 0x7f55c4109450 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:29.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:29.865+0000 7f55c9ed3700 1 -- 172.21.15.146:0/2818423163 >> 172.21.15.146:0/2818423163 conn(0x7f55c40f0b70 msgr2=0x7f55c40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:29.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:29.865+0000 7f55c9ed3700 1 -- 172.21.15.146:0/2818423163 shutdown_connections 2024-01-21T20:06:29.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:29.865+0000 7f55c9ed3700 1 -- 172.21.15.146:0/2818423163 wait complete. 2024-01-21T20:06:29.857 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:06:30.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:29 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:06:29.864480+0000 mon.smithi053 (mon.0) 284 : audit [DBG] from='client.? 172.21.15.146:0/2818423163' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:06:31.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:30 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:06:29.195180+0000 mgr.smithi053.fgsmol (mgr.14164) 226 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:31.237 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:06:31.237 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:06:32.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:32.493+0000 7f86e142e700 1 -- 172.21.15.146:0/1634073748 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86dc0601e0 msgr2=0x7f86dc0f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:32.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:32.493+0000 7f86e142e700 1 --2- 172.21.15.146:0/1634073748 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86dc0601e0 0x7f86dc0f9190 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f86d0009fb0 tx=0x7f86d0009b30).stop 2024-01-21T20:06:32.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:32.493+0000 7f86e142e700 1 -- 172.21.15.146:0/1634073748 shutdown_connections 2024-01-21T20:06:32.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:32.493+0000 7f86e142e700 1 --2- 172.21.15.146:0/1634073748 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86dc0601e0 0x7f86dc0f9190 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:32.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:32.493+0000 7f86e142e700 1 -- 172.21.15.146:0/1634073748 >> 172.21.15.146:0/1634073748 conn(0x7f86dc0f0b70 msgr2=0x7f86dc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:32.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:32.493+0000 7f86e142e700 1 -- 172.21.15.146:0/1634073748 shutdown_connections 2024-01-21T20:06:32.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:32.493+0000 7f86e142e700 1 -- 172.21.15.146:0/1634073748 wait complete. 2024-01-21T20:06:32.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:32.494+0000 7f86e142e700 1 Processor -- start 2024-01-21T20:06:32.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:32.494+0000 7f86e142e700 1 -- start start 2024-01-21T20:06:32.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:32.494+0000 7f86e142e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86dc0601e0 0x7f86dc1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:32.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:32.494+0000 7f86e142e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86dc106bd0 con 0x7f86dc0601e0 2024-01-21T20:06:32.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:32.494+0000 7f86daffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86dc0601e0 0x7f86dc1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:32.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:32.494+0000 7f86daffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86dc0601e0 0x7f86dc1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:56410/0 (socket says 172.21.15.146:56410) 2024-01-21T20:06:32.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:32.494+0000 7f86daffd700 1 -- 172.21.15.146:0/578770514 learned_addr learned my addr 172.21.15.146:0/578770514 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:06:32.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:32.495+0000 7f86daffd700 1 -- 172.21.15.146:0/578770514 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86d0005040 con 0x7f86dc0601e0 2024-01-21T20:06:32.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:32.495+0000 7f86daffd700 1 --2- 172.21.15.146:0/578770514 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86dc0601e0 0x7f86dc1066e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f86d0004dd0 tx=0x7f86d0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:32.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:32.495+0000 7f86c37fe700 1 -- 172.21.15.146:0/578770514 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86d0018070 con 0x7f86dc0601e0 2024-01-21T20:06:32.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:32.495+0000 7f86e142e700 1 -- 172.21.15.146:0/578770514 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86dc106dd0 con 0x7f86dc0601e0 2024-01-21T20:06:32.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:32.495+0000 7f86c37fe700 1 -- 172.21.15.146:0/578770514 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f86d0013410 con 0x7f86dc0601e0 2024-01-21T20:06:32.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:32.495+0000 7f86c37fe700 1 -- 172.21.15.146:0/578770514 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86d001c3c0 con 0x7f86dc0601e0 2024-01-21T20:06:32.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:32.495+0000 7f86e142e700 1 -- 172.21.15.146:0/578770514 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86dc107260 con 0x7f86dc0601e0 2024-01-21T20:06:32.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:32.496+0000 7f86c37fe700 1 -- 172.21.15.146:0/578770514 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f86d001d420 con 0x7f86dc0601e0 2024-01-21T20:06:32.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:32.496+0000 7f86e142e700 1 -- 172.21.15.146:0/578770514 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86dc045ad0 con 0x7f86dc0601e0 2024-01-21T20:06:32.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:32.496+0000 7f86c37fe700 1 --2- 172.21.15.146:0/578770514 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f86c402ff60 0x7f86c4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:32.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:32.496+0000 7f86c37fe700 1 -- 172.21.15.146:0/578770514 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f86d00413d0 con 0x7f86dc0601e0 2024-01-21T20:06:32.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:32.496+0000 7f86da7fc700 1 --2- 172.21.15.146:0/578770514 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f86c402ff60 0x7f86c4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:32.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:32.497+0000 7f86da7fc700 1 --2- 172.21.15.146:0/578770514 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f86c402ff60 0x7f86c4032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f86cc0096a0 tx=0x7f86cc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:32.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:32.501+0000 7f86c37fe700 1 -- 172.21.15.146:0/578770514 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f86d001d820 con 0x7f86dc0601e0 2024-01-21T20:06:32.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:32.704+0000 7f86e142e700 1 -- 172.21.15.146:0/578770514 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f86dc108110 con 0x7f86dc0601e0 2024-01-21T20:06:32.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:32.706+0000 7f86c37fe700 1 -- 172.21.15.146:0/578770514 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f86d001b370 con 0x7f86dc0601e0 2024-01-21T20:06:32.696 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:06:32.696 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:06:32.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:32.706+0000 7f86e142e700 1 -- 172.21.15.146:0/578770514 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f86c402ff60 msgr2=0x7f86c4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:32.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:32.706+0000 7f86e142e700 1 --2- 172.21.15.146:0/578770514 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f86c402ff60 0x7f86c4032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f86cc0096a0 tx=0x7f86cc006b40).stop 2024-01-21T20:06:32.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:32.706+0000 7f86e142e700 1 -- 172.21.15.146:0/578770514 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86dc0601e0 msgr2=0x7f86dc1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:32.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:32.706+0000 7f86e142e700 1 --2- 172.21.15.146:0/578770514 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86dc0601e0 0x7f86dc1066e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f86d0004dd0 tx=0x7f86d0009b30).stop 2024-01-21T20:06:32.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:32.707+0000 7f86e142e700 1 -- 172.21.15.146:0/578770514 shutdown_connections 2024-01-21T20:06:32.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:32.707+0000 7f86e142e700 1 --2- 172.21.15.146:0/578770514 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f86c402ff60 0x7f86c4032420 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:32.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:32.707+0000 7f86e142e700 1 --2- 172.21.15.146:0/578770514 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86dc0601e0 0x7f86dc1066e0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:32.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:32.707+0000 7f86e142e700 1 -- 172.21.15.146:0/578770514 >> 172.21.15.146:0/578770514 conn(0x7f86dc0f0b70 msgr2=0x7f86dc0f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:32.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:32.707+0000 7f86e142e700 1 -- 172.21.15.146:0/578770514 shutdown_connections 2024-01-21T20:06:32.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:32.707+0000 7f86e142e700 1 -- 172.21.15.146:0/578770514 wait complete. 2024-01-21T20:06:32.698 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:06:33.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:32 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:06:31.195453+0000 mgr.smithi053.fgsmol (mgr.14164) 227 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:33.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:32 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:06:32.706231+0000 mon.smithi053 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.146:0/578770514' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:06:34.102 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:06:34.103 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:06:35.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:34 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:06:33.195855+0000 mgr.smithi053.fgsmol (mgr.14164) 228 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:35.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:35.351+0000 7f1832133700 1 -- 172.21.15.146:0/923236534 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f182c0f5950 msgr2=0x7f182c0f5d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:35.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:35.351+0000 7f1832133700 1 --2- 172.21.15.146:0/923236534 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f182c0f5950 0x7f182c0f5d70 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f1820009fb0 tx=0x7f1820009b30).stop 2024-01-21T20:06:35.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:35.351+0000 7f1832133700 1 -- 172.21.15.146:0/923236534 shutdown_connections 2024-01-21T20:06:35.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:35.351+0000 7f1832133700 1 --2- 172.21.15.146:0/923236534 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f182c0f5950 0x7f182c0f5d70 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:35.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:35.351+0000 7f1832133700 1 -- 172.21.15.146:0/923236534 >> 172.21.15.146:0/923236534 conn(0x7f182c0f0260 msgr2=0x7f182c0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:35.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:35.351+0000 7f1832133700 1 -- 172.21.15.146:0/923236534 shutdown_connections 2024-01-21T20:06:35.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:35.351+0000 7f1832133700 1 -- 172.21.15.146:0/923236534 wait complete. 2024-01-21T20:06:35.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:35.352+0000 7f1832133700 1 Processor -- start 2024-01-21T20:06:35.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:35.352+0000 7f1832133700 1 -- start start 2024-01-21T20:06:35.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:35.352+0000 7f1832133700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f182c0f5950 0x7f182c109410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:35.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:35.352+0000 7f1832133700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f182c0f5330 con 0x7f182c0f5950 2024-01-21T20:06:35.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:35.352+0000 7f182b7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f182c0f5950 0x7f182c109410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:35.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:35.352+0000 7f182b7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f182c0f5950 0x7f182c109410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:56430/0 (socket says 172.21.15.146:56430) 2024-01-21T20:06:35.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:35.352+0000 7f182b7fe700 1 -- 172.21.15.146:0/3222434112 learned_addr learned my addr 172.21.15.146:0/3222434112 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:06:35.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:35.352+0000 7f182b7fe700 1 -- 172.21.15.146:0/3222434112 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1820005040 con 0x7f182c0f5950 2024-01-21T20:06:35.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:35.353+0000 7f182b7fe700 1 --2- 172.21.15.146:0/3222434112 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f182c0f5950 0x7f182c109410 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f1820013510 tx=0x7f182000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:35.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:35.353+0000 7f1813fff700 1 -- 172.21.15.146:0/3222434112 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1820009320 con 0x7f182c0f5950 2024-01-21T20:06:35.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:35.353+0000 7f1832133700 1 -- 172.21.15.146:0/3222434112 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f182c0f5530 con 0x7f182c0f5950 2024-01-21T20:06:35.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:35.353+0000 7f1813fff700 1 -- 172.21.15.146:0/3222434112 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1820013f00 con 0x7f182c0f5950 2024-01-21T20:06:35.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:35.353+0000 7f1813fff700 1 -- 172.21.15.146:0/3222434112 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1820014d90 con 0x7f182c0f5950 2024-01-21T20:06:35.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:35.353+0000 7f1832133700 1 -- 172.21.15.146:0/3222434112 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f182c109ac0 con 0x7f182c0f5950 2024-01-21T20:06:35.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:35.354+0000 7f1813fff700 1 -- 172.21.15.146:0/3222434112 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f18200095f0 con 0x7f182c0f5950 2024-01-21T20:06:35.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:35.354+0000 7f1813fff700 1 --2- 172.21.15.146:0/3222434112 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f181402ff60 0x7f1814032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:35.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:35.354+0000 7f1832133700 1 -- 172.21.15.146:0/3222434112 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f182c045ad0 con 0x7f182c0f5950 2024-01-21T20:06:35.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:35.354+0000 7f1813fff700 1 -- 172.21.15.146:0/3222434112 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1820040fc0 con 0x7f182c0f5950 2024-01-21T20:06:35.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:35.354+0000 7f182affd700 1 --2- 172.21.15.146:0/3222434112 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f181402ff60 0x7f1814032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:35.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:35.354+0000 7f182affd700 1 --2- 172.21.15.146:0/3222434112 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f181402ff60 0x7f1814032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f181c0096a0 tx=0x7f181c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:35.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:35.359+0000 7f1813fff700 1 -- 172.21.15.146:0/3222434112 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1820014420 con 0x7f182c0f5950 2024-01-21T20:06:35.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:35.562+0000 7f1832133700 1 -- 172.21.15.146:0/3222434112 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f182c10a2e0 con 0x7f182c0f5950 2024-01-21T20:06:35.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:35.563+0000 7f1813fff700 1 -- 172.21.15.146:0/3222434112 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f182002d090 con 0x7f182c0f5950 2024-01-21T20:06:35.553 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:06:35.553 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:06:35.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:35.564+0000 7f1832133700 1 -- 172.21.15.146:0/3222434112 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f181402ff60 msgr2=0x7f1814032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:35.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:35.564+0000 7f1832133700 1 --2- 172.21.15.146:0/3222434112 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f181402ff60 0x7f1814032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f181c0096a0 tx=0x7f181c006b40).stop 2024-01-21T20:06:35.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:35.564+0000 7f1832133700 1 -- 172.21.15.146:0/3222434112 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f182c0f5950 msgr2=0x7f182c109410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:35.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:35.564+0000 7f1832133700 1 --2- 172.21.15.146:0/3222434112 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f182c0f5950 0x7f182c109410 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f1820013510 tx=0x7f182000b040).stop 2024-01-21T20:06:35.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:35.564+0000 7f1832133700 1 -- 172.21.15.146:0/3222434112 shutdown_connections 2024-01-21T20:06:35.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:35.564+0000 7f1832133700 1 --2- 172.21.15.146:0/3222434112 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f181402ff60 0x7f1814032420 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:35.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:35.564+0000 7f1832133700 1 --2- 172.21.15.146:0/3222434112 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f182c0f5950 0x7f182c109410 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:35.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:35.564+0000 7f1832133700 1 -- 172.21.15.146:0/3222434112 >> 172.21.15.146:0/3222434112 conn(0x7f182c0f0260 msgr2=0x7f182c0f0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:35.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:35.564+0000 7f1832133700 1 -- 172.21.15.146:0/3222434112 shutdown_connections 2024-01-21T20:06:35.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:35.564+0000 7f1832133700 1 -- 172.21.15.146:0/3222434112 wait complete. 2024-01-21T20:06:35.556 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:06:36.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:35 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:06:35.564115+0000 mon.smithi053 (mon.0) 286 : audit [DBG] from='client.? 172.21.15.146:0/3222434112' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:06:36.936 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:06:36.936 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:06:37.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:36 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:06:35.196316+0000 mgr.smithi053.fgsmol (mgr.14164) 229 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:38.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:38.184+0000 7f992291d700 1 -- 172.21.15.146:0/4207620336 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f991c0f78e0 msgr2=0x7f991c0f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:38.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:38.184+0000 7f992291d700 1 --2- 172.21.15.146:0/4207620336 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f991c0f78e0 0x7f991c0f7d00 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f9918009fb0 tx=0x7f9918009b30).stop 2024-01-21T20:06:38.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:38.184+0000 7f992291d700 1 -- 172.21.15.146:0/4207620336 shutdown_connections 2024-01-21T20:06:38.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:38.184+0000 7f992291d700 1 --2- 172.21.15.146:0/4207620336 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f991c0f78e0 0x7f991c0f7d00 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:38.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:38.184+0000 7f992291d700 1 -- 172.21.15.146:0/4207620336 >> 172.21.15.146:0/4207620336 conn(0x7f991c0f0120 msgr2=0x7f991c0f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:38.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:38.184+0000 7f992291d700 1 -- 172.21.15.146:0/4207620336 shutdown_connections 2024-01-21T20:06:38.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:38.184+0000 7f992291d700 1 -- 172.21.15.146:0/4207620336 wait complete. 2024-01-21T20:06:38.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:38.184+0000 7f992291d700 1 Processor -- start 2024-01-21T20:06:38.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:38.184+0000 7f992291d700 1 -- start start 2024-01-21T20:06:38.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:38.185+0000 7f992291d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f991c0f78e0 0x7f991c0f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:38.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:38.185+0000 7f992291d700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f991c0f5630 con 0x7f991c0f78e0 2024-01-21T20:06:38.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:38.185+0000 7f992191b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f991c0f78e0 0x7f991c0f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:38.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:38.185+0000 7f992191b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f991c0f78e0 0x7f991c0f71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:40096/0 (socket says 172.21.15.146:40096) 2024-01-21T20:06:38.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:38.185+0000 7f992191b700 1 -- 172.21.15.146:0/770762542 learned_addr learned my addr 172.21.15.146:0/770762542 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:06:38.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:38.185+0000 7f992191b700 1 -- 172.21.15.146:0/770762542 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9918005040 con 0x7f991c0f78e0 2024-01-21T20:06:38.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:38.185+0000 7f992191b700 1 --2- 172.21.15.146:0/770762542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f991c0f78e0 0x7f991c0f71b0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f9918013510 tx=0x7f991800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:38.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:38.185+0000 7f99127fc700 1 -- 172.21.15.146:0/770762542 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9918009320 con 0x7f991c0f78e0 2024-01-21T20:06:38.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:38.186+0000 7f992291d700 1 -- 172.21.15.146:0/770762542 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f991c0f57d0 con 0x7f991c0f78e0 2024-01-21T20:06:38.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:38.186+0000 7f99127fc700 1 -- 172.21.15.146:0/770762542 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9918013f00 con 0x7f991c0f78e0 2024-01-21T20:06:38.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:38.186+0000 7f99127fc700 1 -- 172.21.15.146:0/770762542 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9918014dc0 con 0x7f991c0f78e0 2024-01-21T20:06:38.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:38.186+0000 7f992291d700 1 -- 172.21.15.146:0/770762542 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f991c0f5c60 con 0x7f991c0f78e0 2024-01-21T20:06:38.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:38.186+0000 7f99127fc700 1 -- 172.21.15.146:0/770762542 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f99180095f0 con 0x7f991c0f78e0 2024-01-21T20:06:38.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:38.186+0000 7f992291d700 1 -- 172.21.15.146:0/770762542 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f991c059100 con 0x7f991c0f78e0 2024-01-21T20:06:38.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:38.186+0000 7f99127fc700 1 --2- 172.21.15.146:0/770762542 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f990802ff60 0x7f9908032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:38.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:38.186+0000 7f99127fc700 1 -- 172.21.15.146:0/770762542 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9918041000 con 0x7f991c0f78e0 2024-01-21T20:06:38.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:38.187+0000 7f992111a700 1 --2- 172.21.15.146:0/770762542 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f990802ff60 0x7f9908032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:38.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:38.187+0000 7f992111a700 1 --2- 172.21.15.146:0/770762542 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f990802ff60 0x7f9908032420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f990c0096a0 tx=0x7f990c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:38.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:38.191+0000 7f99127fc700 1 -- 172.21.15.146:0/770762542 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9918014420 con 0x7f991c0f78e0 2024-01-21T20:06:38.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:38.396+0000 7f992291d700 1 -- 172.21.15.146:0/770762542 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f991c0fbce0 con 0x7f991c0f78e0 2024-01-21T20:06:38.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:38.398+0000 7f99127fc700 1 -- 172.21.15.146:0/770762542 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9918010230 con 0x7f991c0f78e0 2024-01-21T20:06:38.388 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:06:38.388 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:06:38.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:38.398+0000 7f992291d700 1 -- 172.21.15.146:0/770762542 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f990802ff60 msgr2=0x7f9908032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:38.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:38.398+0000 7f992291d700 1 --2- 172.21.15.146:0/770762542 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f990802ff60 0x7f9908032420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f990c0096a0 tx=0x7f990c006b40).stop 2024-01-21T20:06:38.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:38.398+0000 7f992291d700 1 -- 172.21.15.146:0/770762542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f991c0f78e0 msgr2=0x7f991c0f71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:38.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:38.398+0000 7f992291d700 1 --2- 172.21.15.146:0/770762542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f991c0f78e0 0x7f991c0f71b0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f9918013510 tx=0x7f991800b040).stop 2024-01-21T20:06:38.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:38.398+0000 7f992291d700 1 -- 172.21.15.146:0/770762542 shutdown_connections 2024-01-21T20:06:38.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:38.398+0000 7f992291d700 1 --2- 172.21.15.146:0/770762542 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f990802ff60 0x7f9908032420 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:38.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:38.398+0000 7f992291d700 1 --2- 172.21.15.146:0/770762542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f991c0f78e0 0x7f991c0f71b0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:38.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:38.398+0000 7f992291d700 1 -- 172.21.15.146:0/770762542 >> 172.21.15.146:0/770762542 conn(0x7f991c0f0120 msgr2=0x7f991c0f0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:38.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:38.398+0000 7f992291d700 1 -- 172.21.15.146:0/770762542 shutdown_connections 2024-01-21T20:06:38.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:38.398+0000 7f992291d700 1 -- 172.21.15.146:0/770762542 wait complete. 2024-01-21T20:06:38.390 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:06:39.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:38 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:06:37.196573+0000 mgr.smithi053.fgsmol (mgr.14164) 230 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:39.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:38 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:06:38.398113+0000 mon.smithi053 (mon.0) 287 : audit [DBG] from='client.? 172.21.15.146:0/770762542' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:06:39.770 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:06:39.770 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:06:41.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:41.027+0000 7feac4ac4700 1 -- 172.21.15.146:0/4214121114 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feac0060280 msgr2=0x7feac00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:41.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:41.027+0000 7feac4ac4700 1 --2- 172.21.15.146:0/4214121114 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feac0060280 0x7feac00f90f0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7feab4009fb0 tx=0x7feab4009b30).stop 2024-01-21T20:06:41.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:41.027+0000 7feac4ac4700 1 -- 172.21.15.146:0/4214121114 shutdown_connections 2024-01-21T20:06:41.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:41.027+0000 7feac4ac4700 1 --2- 172.21.15.146:0/4214121114 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feac0060280 0x7feac00f90f0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:41.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:41.027+0000 7feac4ac4700 1 -- 172.21.15.146:0/4214121114 >> 172.21.15.146:0/4214121114 conn(0x7feac00f0b70 msgr2=0x7feac00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:41.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:41.027+0000 7feac4ac4700 1 -- 172.21.15.146:0/4214121114 shutdown_connections 2024-01-21T20:06:41.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:41.027+0000 7feac4ac4700 1 -- 172.21.15.146:0/4214121114 wait complete. 2024-01-21T20:06:41.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:41.028+0000 7feac4ac4700 1 Processor -- start 2024-01-21T20:06:41.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:41.028+0000 7feac4ac4700 1 -- start start 2024-01-21T20:06:41.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:41.028+0000 7feac4ac4700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feac0060280 0x7feac00f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:41.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:41.028+0000 7feac4ac4700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac00f4ff0 con 0x7feac0060280 2024-01-21T20:06:41.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:41.028+0000 7feabe59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feac0060280 0x7feac00f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:41.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:41.028+0000 7feabe59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feac0060280 0x7feac00f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:40122/0 (socket says 172.21.15.146:40122) 2024-01-21T20:06:41.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:41.028+0000 7feabe59c700 1 -- 172.21.15.146:0/2087358312 learned_addr learned my addr 172.21.15.146:0/2087358312 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:06:41.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:41.028+0000 7feabe59c700 1 -- 172.21.15.146:0/2087358312 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab4005040 con 0x7feac0060280 2024-01-21T20:06:41.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:41.029+0000 7feabe59c700 1 --2- 172.21.15.146:0/2087358312 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feac0060280 0x7feac00f6b70 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7feab4009830 tx=0x7feab400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:41.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:41.029+0000 7feaaeffd700 1 -- 172.21.15.146:0/2087358312 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feab4009320 con 0x7feac0060280 2024-01-21T20:06:41.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:41.029+0000 7feac4ac4700 1 -- 172.21.15.146:0/2087358312 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feac00f5190 con 0x7feac0060280 2024-01-21T20:06:41.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:41.029+0000 7feaaeffd700 1 -- 172.21.15.146:0/2087358312 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feab40136c0 con 0x7feac0060280 2024-01-21T20:06:41.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:41.029+0000 7feaaeffd700 1 -- 172.21.15.146:0/2087358312 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feab401ee90 con 0x7feac0060280 2024-01-21T20:06:41.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:41.029+0000 7feac4ac4700 1 -- 172.21.15.146:0/2087358312 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feac00f5620 con 0x7feac0060280 2024-01-21T20:06:41.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:41.029+0000 7feaaeffd700 1 -- 172.21.15.146:0/2087358312 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feab4014440 con 0x7feac0060280 2024-01-21T20:06:41.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:41.030+0000 7feaaeffd700 1 --2- 172.21.15.146:0/2087358312 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7feaa802ff10 0x7feaa80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:41.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:41.030+0000 7feac4ac4700 1 -- 172.21.15.146:0/2087358312 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feac0059100 con 0x7feac0060280 2024-01-21T20:06:41.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:41.030+0000 7feaaeffd700 1 -- 172.21.15.146:0/2087358312 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feab40412d0 con 0x7feac0060280 2024-01-21T20:06:41.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:41.030+0000 7feabdd9b700 1 --2- 172.21.15.146:0/2087358312 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7feaa802ff10 0x7feaa80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:41.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:41.030+0000 7feabdd9b700 1 --2- 172.21.15.146:0/2087358312 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7feaa802ff10 0x7feaa80323d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7feab00096a0 tx=0x7feab0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:41.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:41.034+0000 7feaaeffd700 1 -- 172.21.15.146:0/2087358312 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feab401e420 con 0x7feac0060280 2024-01-21T20:06:41.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:40 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:06:39.196933+0000 mgr.smithi053.fgsmol (mgr.14164) 231 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:41.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:41.239+0000 7feac4ac4700 1 -- 172.21.15.146:0/2087358312 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feac00fbf50 con 0x7feac0060280 2024-01-21T20:06:41.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:41.240+0000 7feaaeffd700 1 -- 172.21.15.146:0/2087358312 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7feab401c320 con 0x7feac0060280 2024-01-21T20:06:41.230 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:06:41.230 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:06:41.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:41.241+0000 7feac4ac4700 1 -- 172.21.15.146:0/2087358312 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7feaa802ff10 msgr2=0x7feaa80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:41.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:41.241+0000 7feac4ac4700 1 --2- 172.21.15.146:0/2087358312 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7feaa802ff10 0x7feaa80323d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7feab00096a0 tx=0x7feab0006b40).stop 2024-01-21T20:06:41.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:41.241+0000 7feac4ac4700 1 -- 172.21.15.146:0/2087358312 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feac0060280 msgr2=0x7feac00f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:41.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:41.241+0000 7feac4ac4700 1 --2- 172.21.15.146:0/2087358312 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feac0060280 0x7feac00f6b70 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7feab4009830 tx=0x7feab400b040).stop 2024-01-21T20:06:41.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:41.241+0000 7feac4ac4700 1 -- 172.21.15.146:0/2087358312 shutdown_connections 2024-01-21T20:06:41.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:41.241+0000 7feac4ac4700 1 --2- 172.21.15.146:0/2087358312 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7feaa802ff10 0x7feaa80323d0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:41.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:41.241+0000 7feac4ac4700 1 --2- 172.21.15.146:0/2087358312 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feac0060280 0x7feac00f6b70 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:41.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:41.241+0000 7feac4ac4700 1 -- 172.21.15.146:0/2087358312 >> 172.21.15.146:0/2087358312 conn(0x7feac00f0b70 msgr2=0x7feac00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:41.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:41.241+0000 7feac4ac4700 1 -- 172.21.15.146:0/2087358312 shutdown_connections 2024-01-21T20:06:41.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:41.242+0000 7feac4ac4700 1 -- 172.21.15.146:0/2087358312 wait complete. 2024-01-21T20:06:41.232 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:06:42.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cephadm 2024-01-21T20:06:40.484084+0000 mgr.smithi053.fgsmol (mgr.14164) 232 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi053 ... 2024-01-21T20:06:42.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: Verifying port 9100 ... 2024-01-21T20:06:42.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: Non-zero exit code 1 from systemctl start ceph-57f6722c-b897-11ee-95b0-87774f69a715@node-exporter.smithi053 2024-01-21T20:06:42.184 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: systemctl: stderr Job for ceph-57f6722c-b897-11ee-95b0-87774f69a715@node-exporter.smithi053.service failed because a timeout was exceeded. 2024-01-21T20:06:42.184 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: systemctl: stderr See "systemctl status ceph-57f6722c-b897-11ee-95b0-87774f69a715@node-exporter.smithi053.service" and "journalctl -xe" for details. 2024-01-21T20:06:42.184 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: Traceback (most recent call last): 2024-01-21T20:06:42.184 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: File "/var/lib/ceph/57f6722c-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T20:06:42.184 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: main() 2024-01-21T20:06:42.184 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: File "/var/lib/ceph/57f6722c-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T20:06:42.184 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: r = ctx.func(ctx) 2024-01-21T20:06:42.184 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: File "/var/lib/ceph/57f6722c-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T20:06:42.184 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: return func(ctx) 2024-01-21T20:06:42.184 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: File "/var/lib/ceph/57f6722c-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-01-21T20:06:42.184 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: ports=daemon_ports) 2024-01-21T20:06:42.184 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: File "/var/lib/ceph/57f6722c-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-01-21T20:06:42.184 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: c, osd_fsid=osd_fsid, ports=ports) 2024-01-21T20:06:42.184 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: File "/var/lib/ceph/57f6722c-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-01-21T20:06:42.185 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-01-21T20:06:42.185 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: File "/var/lib/ceph/57f6722c-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-01-21T20:06:42.185 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-21T20:06:42.185 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: RuntimeError: Failed command: systemctl start ceph-57f6722c-b897-11ee-95b0-87774f69a715@node-exporter.smithi053 2024-01-21T20:06:42.185 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: Traceback (most recent call last): 2024-01-21T20:06:42.185 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-01-21T20:06:42.185 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: yield (conn, connr) 2024-01-21T20:06:42.185 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-01-21T20:06:42.185 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: code, '\n'.join(err))) 2024-01-21T20:06:42.185 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi053 ... 2024-01-21T20:06:42.185 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: Verifying port 9100 ... 2024-01-21T20:06:42.185 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: Non-zero exit code 1 from systemctl start ceph-57f6722c-b897-11ee-95b0-87774f69a715@node-exporter.smithi053 2024-01-21T20:06:42.185 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: systemctl: stderr Job for ceph-57f6722c-b897-11ee-95b0-87774f69a715@node-exporter.smithi053.service failed because a timeout was exceeded. 2024-01-21T20:06:42.185 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: systemctl: stderr See "systemctl status ceph-57f6722c-b897-11ee-95b0-87774f69a715@node-exporter.smithi053.service" and "journalctl -xe" for details. 2024-01-21T20:06:42.185 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: Traceback (most recent call last): 2024-01-21T20:06:42.185 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: File "/var/lib/ceph/57f6722c-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T20:06:42.186 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: main() 2024-01-21T20:06:42.186 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: File "/var/lib/ceph/57f6722c-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T20:06:42.186 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: r = ctx.func(ctx) 2024-01-21T20:06:42.186 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: File "/var/lib/ceph/57f6722c-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T20:06:42.186 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: return func(ctx) 2024-01-21T20:06:42.186 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: File "/var/lib/ceph/57f6722c-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-01-21T20:06:42.186 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: ports=daemon_ports) 2024-01-21T20:06:42.186 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: File "/var/lib/ceph/57f6722c-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-01-21T20:06:42.186 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: c, osd_fsid=osd_fsid, ports=ports) 2024-01-21T20:06:42.186 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: File "/var/lib/ceph/57f6722c-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-01-21T20:06:42.186 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-01-21T20:06:42.186 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: File "/var/lib/ceph/57f6722c-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-01-21T20:06:42.186 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-21T20:06:42.186 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: RuntimeError: Failed command: systemctl start ceph-57f6722c-b897-11ee-95b0-87774f69a715@node-exporter.smithi053 2024-01-21T20:06:42.186 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cephadm 2024-01-21T20:06:40.490111+0000 mgr.smithi053.fgsmol (mgr.14164) 233 : cephadm [INF] Deploying daemon prometheus.smithi053 on smithi053 2024-01-21T20:06:42.187 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:41 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:06:41.240027+0000 mon.smithi053 (mon.0) 288 : audit [DBG] from='client.? 172.21.15.146:0/2087358312' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:06:42.623 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:06:42.624 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:06:43.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:42 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:06:41.197351+0000 mgr.smithi053.fgsmol (mgr.14164) 234 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:43.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:43.881+0000 7fabb491d700 1 -- 172.21.15.146:0/2215477337 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabb00f5410 msgr2=0x7fabb00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:43.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:43.881+0000 7fabb491d700 1 --2- 172.21.15.146:0/2215477337 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabb00f5410 0x7fabb00f5830 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7faba4009fb0 tx=0x7faba4009b30).stop 2024-01-21T20:06:43.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:43.881+0000 7fabb491d700 1 -- 172.21.15.146:0/2215477337 shutdown_connections 2024-01-21T20:06:43.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:43.881+0000 7fabb491d700 1 --2- 172.21.15.146:0/2215477337 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabb00f5410 0x7fabb00f5830 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:43.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:43.881+0000 7fabb491d700 1 -- 172.21.15.146:0/2215477337 >> 172.21.15.146:0/2215477337 conn(0x7fabb00f0b70 msgr2=0x7fabb00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:43.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:43.881+0000 7fabb491d700 1 -- 172.21.15.146:0/2215477337 shutdown_connections 2024-01-21T20:06:43.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:43.881+0000 7fabb491d700 1 -- 172.21.15.146:0/2215477337 wait complete. 2024-01-21T20:06:43.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:43.882+0000 7fabb491d700 1 Processor -- start 2024-01-21T20:06:43.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:43.882+0000 7fabb491d700 1 -- start start 2024-01-21T20:06:43.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:43.882+0000 7fabb491d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabb00f5410 0x7fabb01055d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:43.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:43.882+0000 7fabb491d700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb0105ac0 con 0x7fabb00f5410 2024-01-21T20:06:43.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:43.882+0000 7fabae59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabb00f5410 0x7fabb01055d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:43.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:43.882+0000 7fabae59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabb00f5410 0x7fabb01055d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:40148/0 (socket says 172.21.15.146:40148) 2024-01-21T20:06:43.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:43.882+0000 7fabae59c700 1 -- 172.21.15.146:0/3662746779 learned_addr learned my addr 172.21.15.146:0/3662746779 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:06:43.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:43.883+0000 7fabae59c700 1 -- 172.21.15.146:0/3662746779 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faba4005040 con 0x7fabb00f5410 2024-01-21T20:06:43.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:43.883+0000 7fabae59c700 1 --2- 172.21.15.146:0/3662746779 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabb00f5410 0x7fabb01055d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7faba4013510 tx=0x7faba400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:43.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:43.883+0000 7fab9effd700 1 -- 172.21.15.146:0/3662746779 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faba4009320 con 0x7fabb00f5410 2024-01-21T20:06:43.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:43.883+0000 7fabb491d700 1 -- 172.21.15.146:0/3662746779 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabb0105cc0 con 0x7fabb00f5410 2024-01-21T20:06:43.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:43.883+0000 7fab9effd700 1 -- 172.21.15.146:0/3662746779 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faba4013f00 con 0x7fabb00f5410 2024-01-21T20:06:43.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:43.883+0000 7fab9effd700 1 -- 172.21.15.146:0/3662746779 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faba4014d90 con 0x7fabb00f5410 2024-01-21T20:06:43.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:43.883+0000 7fabb491d700 1 -- 172.21.15.146:0/3662746779 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabb0106150 con 0x7fabb00f5410 2024-01-21T20:06:43.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:43.884+0000 7fab9effd700 1 -- 172.21.15.146:0/3662746779 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faba40095f0 con 0x7fabb00f5410 2024-01-21T20:06:43.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:43.884+0000 7fabb491d700 1 -- 172.21.15.146:0/3662746779 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabb0045ad0 con 0x7fabb00f5410 2024-01-21T20:06:43.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:43.884+0000 7fab9effd700 1 --2- 172.21.15.146:0/3662746779 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fab9802ff60 0x7fab98032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:43.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:43.884+0000 7fab9effd700 1 -- 172.21.15.146:0/3662746779 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faba4041000 con 0x7fabb00f5410 2024-01-21T20:06:43.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:43.884+0000 7fabadd9b700 1 --2- 172.21.15.146:0/3662746779 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fab9802ff60 0x7fab98032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:43.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:43.885+0000 7fabadd9b700 1 --2- 172.21.15.146:0/3662746779 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fab9802ff60 0x7fab98032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7faba00096a0 tx=0x7faba0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:43.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:43.889+0000 7fab9effd700 1 -- 172.21.15.146:0/3662746779 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faba4014420 con 0x7fabb00f5410 2024-01-21T20:06:44.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:44.093+0000 7fabb491d700 1 -- 172.21.15.146:0/3662746779 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fabb0107000 con 0x7fabb00f5410 2024-01-21T20:06:44.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:44.094+0000 7fab9effd700 1 -- 172.21.15.146:0/3662746779 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7faba4010230 con 0x7fabb00f5410 2024-01-21T20:06:44.085 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:06:44.085 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:06:44.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:44.096+0000 7fabb491d700 1 -- 172.21.15.146:0/3662746779 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fab9802ff60 msgr2=0x7fab98032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:44.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:44.096+0000 7fabb491d700 1 --2- 172.21.15.146:0/3662746779 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fab9802ff60 0x7fab98032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7faba00096a0 tx=0x7faba0006b40).stop 2024-01-21T20:06:44.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:44.096+0000 7fabb491d700 1 -- 172.21.15.146:0/3662746779 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabb00f5410 msgr2=0x7fabb01055d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:44.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:44.096+0000 7fabb491d700 1 --2- 172.21.15.146:0/3662746779 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabb00f5410 0x7fabb01055d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7faba4013510 tx=0x7faba400b040).stop 2024-01-21T20:06:44.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:44.096+0000 7fabb491d700 1 -- 172.21.15.146:0/3662746779 shutdown_connections 2024-01-21T20:06:44.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:44.096+0000 7fabb491d700 1 --2- 172.21.15.146:0/3662746779 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fab9802ff60 0x7fab98032420 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:44.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:44.096+0000 7fabb491d700 1 --2- 172.21.15.146:0/3662746779 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabb00f5410 0x7fabb01055d0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:44.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:44.096+0000 7fabb491d700 1 -- 172.21.15.146:0/3662746779 >> 172.21.15.146:0/3662746779 conn(0x7fabb00f0b70 msgr2=0x7fabb00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:44.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:44.096+0000 7fabb491d700 1 -- 172.21.15.146:0/3662746779 shutdown_connections 2024-01-21T20:06:44.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:44.096+0000 7fabb491d700 1 -- 172.21.15.146:0/3662746779 wait complete. 2024-01-21T20:06:44.087 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:06:45.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:44 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:06:43.197761+0000 mgr.smithi053.fgsmol (mgr.14164) 235 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:45.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:44 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:06:44.094697+0000 mon.smithi053 (mon.0) 289 : audit [DBG] from='client.? 172.21.15.146:0/3662746779' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:06:45.458 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:06:45.458 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:06:46.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:46.728+0000 7f60688ac700 1 -- 172.21.15.146:0/1192714081 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60640f5410 msgr2=0x7f60640f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:46.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:46.728+0000 7f60688ac700 1 --2- 172.21.15.146:0/1192714081 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60640f5410 0x7f60640f5830 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f6058009fb0 tx=0x7f6058009b30).stop 2024-01-21T20:06:46.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:46.728+0000 7f60688ac700 1 -- 172.21.15.146:0/1192714081 shutdown_connections 2024-01-21T20:06:46.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:46.728+0000 7f60688ac700 1 --2- 172.21.15.146:0/1192714081 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60640f5410 0x7f60640f5830 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:46.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:46.728+0000 7f60688ac700 1 -- 172.21.15.146:0/1192714081 >> 172.21.15.146:0/1192714081 conn(0x7f60640f0b70 msgr2=0x7f60640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:46.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:46.728+0000 7f60688ac700 1 -- 172.21.15.146:0/1192714081 shutdown_connections 2024-01-21T20:06:46.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:46.728+0000 7f60688ac700 1 -- 172.21.15.146:0/1192714081 wait complete. 2024-01-21T20:06:46.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:46.728+0000 7f60688ac700 1 Processor -- start 2024-01-21T20:06:46.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:46.728+0000 7f60688ac700 1 -- start start 2024-01-21T20:06:46.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:46.729+0000 7f60688ac700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60640f5410 0x7f60641093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:46.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:46.729+0000 7f60688ac700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6064060280 con 0x7f60640f5410 2024-01-21T20:06:46.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:46.729+0000 7f606259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60640f5410 0x7f60641093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:46.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:46.729+0000 7f606259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60640f5410 0x7f60641093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:40178/0 (socket says 172.21.15.146:40178) 2024-01-21T20:06:46.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:46.729+0000 7f606259c700 1 -- 172.21.15.146:0/4089892789 learned_addr learned my addr 172.21.15.146:0/4089892789 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:06:46.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:46.729+0000 7f606259c700 1 -- 172.21.15.146:0/4089892789 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6058005040 con 0x7f60640f5410 2024-01-21T20:06:46.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:46.729+0000 7f606259c700 1 --2- 172.21.15.146:0/4089892789 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60640f5410 0x7f60641093f0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f6058013510 tx=0x7f605800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:46.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:46.730+0000 7f6052ffd700 1 -- 172.21.15.146:0/4089892789 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6058009320 con 0x7f60640f5410 2024-01-21T20:06:46.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:46.730+0000 7f60688ac700 1 -- 172.21.15.146:0/4089892789 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6064060480 con 0x7f60640f5410 2024-01-21T20:06:46.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:46.730+0000 7f6052ffd700 1 -- 172.21.15.146:0/4089892789 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6058013f00 con 0x7f60640f5410 2024-01-21T20:06:46.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:46.730+0000 7f6052ffd700 1 -- 172.21.15.146:0/4089892789 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6058014d90 con 0x7f60640f5410 2024-01-21T20:06:46.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:46.730+0000 7f60688ac700 1 -- 172.21.15.146:0/4089892789 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6064109b20 con 0x7f60640f5410 2024-01-21T20:06:46.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:46.731+0000 7f6052ffd700 1 -- 172.21.15.146:0/4089892789 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f60580095f0 con 0x7f60640f5410 2024-01-21T20:06:46.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:46.731+0000 7f60688ac700 1 -- 172.21.15.146:0/4089892789 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6064045ad0 con 0x7f60640f5410 2024-01-21T20:06:46.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:46.731+0000 7f6052ffd700 1 --2- 172.21.15.146:0/4089892789 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f604c02ff60 0x7f604c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:46.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:46.731+0000 7f6052ffd700 1 -- 172.21.15.146:0/4089892789 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6058041000 con 0x7f60640f5410 2024-01-21T20:06:46.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:46.731+0000 7f6061d9b700 1 --2- 172.21.15.146:0/4089892789 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f604c02ff60 0x7f604c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:46.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:46.731+0000 7f6061d9b700 1 --2- 172.21.15.146:0/4089892789 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f604c02ff60 0x7f604c032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f60540096a0 tx=0x7f6054006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:46.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:46.736+0000 7f6052ffd700 1 -- 172.21.15.146:0/4089892789 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6058014420 con 0x7f60640f5410 2024-01-21T20:06:46.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:46.941+0000 7f60688ac700 1 -- 172.21.15.146:0/4089892789 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f606410a340 con 0x7f60640f5410 2024-01-21T20:06:46.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:46.942+0000 7f6052ffd700 1 -- 172.21.15.146:0/4089892789 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f605801d7f0 con 0x7f60640f5410 2024-01-21T20:06:46.932 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:06:46.932 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:06:46.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:46.944+0000 7f60688ac700 1 -- 172.21.15.146:0/4089892789 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f604c02ff60 msgr2=0x7f604c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:46.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:46.944+0000 7f60688ac700 1 --2- 172.21.15.146:0/4089892789 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f604c02ff60 0x7f604c032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f60540096a0 tx=0x7f6054006b40).stop 2024-01-21T20:06:46.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:46.944+0000 7f60688ac700 1 -- 172.21.15.146:0/4089892789 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60640f5410 msgr2=0x7f60641093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:46.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:46.944+0000 7f60688ac700 1 --2- 172.21.15.146:0/4089892789 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60640f5410 0x7f60641093f0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f6058013510 tx=0x7f605800b040).stop 2024-01-21T20:06:46.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:46.944+0000 7f60688ac700 1 -- 172.21.15.146:0/4089892789 shutdown_connections 2024-01-21T20:06:46.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:46.944+0000 7f60688ac700 1 --2- 172.21.15.146:0/4089892789 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f604c02ff60 0x7f604c032420 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:46.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:46.944+0000 7f60688ac700 1 --2- 172.21.15.146:0/4089892789 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60640f5410 0x7f60641093f0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:46.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:46.944+0000 7f60688ac700 1 -- 172.21.15.146:0/4089892789 >> 172.21.15.146:0/4089892789 conn(0x7f60640f0b70 msgr2=0x7f60640f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:46.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:46.944+0000 7f60688ac700 1 -- 172.21.15.146:0/4089892789 shutdown_connections 2024-01-21T20:06:46.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:46.944+0000 7f60688ac700 1 -- 172.21.15.146:0/4089892789 wait complete. 2024-01-21T20:06:46.935 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:06:47.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:46 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:06:45.198302+0000 mgr.smithi053.fgsmol (mgr.14164) 236 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:48.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:47 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:06:46.942502+0000 mon.smithi053 (mon.0) 290 : audit [DBG] from='client.? 172.21.15.146:0/4089892789' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:06:48.326 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:06:48.326 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:06:49.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:48 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:06:47.198722+0000 mgr.smithi053.fgsmol (mgr.14164) 237 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:49.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:49.560+0000 7fa7f7fff700 1 -- 172.21.15.146:0/677550814 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa7f80601e0 msgr2=0x7fa7f80f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:49.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:49.560+0000 7fa7f7fff700 1 --2- 172.21.15.146:0/677550814 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa7f80601e0 0x7fa7f80f8f20 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7fa7ec009fb0 tx=0x7fa7ec009b30).stop 2024-01-21T20:06:49.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:49.560+0000 7fa7f7fff700 1 -- 172.21.15.146:0/677550814 shutdown_connections 2024-01-21T20:06:49.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:49.560+0000 7fa7f7fff700 1 --2- 172.21.15.146:0/677550814 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa7f80601e0 0x7fa7f80f8f20 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:49.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:49.560+0000 7fa7f7fff700 1 -- 172.21.15.146:0/677550814 >> 172.21.15.146:0/677550814 conn(0x7fa7f80f09a0 msgr2=0x7fa7f80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:49.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:49.560+0000 7fa7f7fff700 1 -- 172.21.15.146:0/677550814 shutdown_connections 2024-01-21T20:06:49.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:49.560+0000 7fa7f7fff700 1 -- 172.21.15.146:0/677550814 wait complete. 2024-01-21T20:06:49.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:49.560+0000 7fa7f7fff700 1 Processor -- start 2024-01-21T20:06:49.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:49.560+0000 7fa7f7fff700 1 -- start start 2024-01-21T20:06:49.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:49.561+0000 7fa7f7fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa7f80601e0 0x7fa7f8106520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:49.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:49.561+0000 7fa7f7fff700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f8106a10 con 0x7fa7f80601e0 2024-01-21T20:06:49.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:49.561+0000 7fa7f6ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa7f80601e0 0x7fa7f8106520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:49.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:49.561+0000 7fa7f6ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa7f80601e0 0x7fa7f8106520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:42548/0 (socket says 172.21.15.146:42548) 2024-01-21T20:06:49.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:49.561+0000 7fa7f6ffd700 1 -- 172.21.15.146:0/1250576574 learned_addr learned my addr 172.21.15.146:0/1250576574 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:06:49.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:49.561+0000 7fa7f6ffd700 1 -- 172.21.15.146:0/1250576574 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7ec005040 con 0x7fa7f80601e0 2024-01-21T20:06:49.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:49.561+0000 7fa7f6ffd700 1 --2- 172.21.15.146:0/1250576574 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa7f80601e0 0x7fa7f8106520 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7fa7ec014000 tx=0x7fa7ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:49.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:49.561+0000 7fa7df7fe700 1 -- 172.21.15.146:0/1250576574 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7ec0095f0 con 0x7fa7f80601e0 2024-01-21T20:06:49.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:49.562+0000 7fa7f7fff700 1 -- 172.21.15.146:0/1250576574 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7f8106c10 con 0x7fa7f80601e0 2024-01-21T20:06:49.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:49.562+0000 7fa7df7fe700 1 -- 172.21.15.146:0/1250576574 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa7ec007e70 con 0x7fa7f80601e0 2024-01-21T20:06:49.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:49.562+0000 7fa7df7fe700 1 -- 172.21.15.146:0/1250576574 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7ec014ea0 con 0x7fa7f80601e0 2024-01-21T20:06:49.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:49.562+0000 7fa7f7fff700 1 -- 172.21.15.146:0/1250576574 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7f81070a0 con 0x7fa7f80601e0 2024-01-21T20:06:49.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:49.562+0000 7fa7df7fe700 1 -- 172.21.15.146:0/1250576574 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa7ec025070 con 0x7fa7f80601e0 2024-01-21T20:06:49.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:49.562+0000 7fa7f7fff700 1 -- 172.21.15.146:0/1250576574 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7f8045ad0 con 0x7fa7f80601e0 2024-01-21T20:06:49.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:49.562+0000 7fa7df7fe700 1 --2- 172.21.15.146:0/1250576574 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa7e002ff60 0x7fa7e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:49.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:49.563+0000 7fa7df7fe700 1 -- 172.21.15.146:0/1250576574 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa7ec0409f0 con 0x7fa7f80601e0 2024-01-21T20:06:49.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:49.563+0000 7fa7f67fc700 1 --2- 172.21.15.146:0/1250576574 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa7e002ff60 0x7fa7e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:49.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:49.563+0000 7fa7f67fc700 1 --2- 172.21.15.146:0/1250576574 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa7e002ff60 0x7fa7e0032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fa7e80096a0 tx=0x7fa7e8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:49.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:49.567+0000 7fa7df7fe700 1 -- 172.21.15.146:0/1250576574 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa7ec00f4e0 con 0x7fa7f80601e0 2024-01-21T20:06:49.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:49.771+0000 7fa7f7fff700 1 -- 172.21.15.146:0/1250576574 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa7f8107f50 con 0x7fa7f80601e0 2024-01-21T20:06:49.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:49.771+0000 7fa7df7fe700 1 -- 172.21.15.146:0/1250576574 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa7ec00f300 con 0x7fa7f80601e0 2024-01-21T20:06:49.761 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:06:49.761 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:06:49.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:49.773+0000 7fa7f7fff700 1 -- 172.21.15.146:0/1250576574 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa7e002ff60 msgr2=0x7fa7e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:49.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:49.773+0000 7fa7f7fff700 1 --2- 172.21.15.146:0/1250576574 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa7e002ff60 0x7fa7e0032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fa7e80096a0 tx=0x7fa7e8006b40).stop 2024-01-21T20:06:49.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:49.773+0000 7fa7f7fff700 1 -- 172.21.15.146:0/1250576574 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa7f80601e0 msgr2=0x7fa7f8106520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:49.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:49.773+0000 7fa7f7fff700 1 --2- 172.21.15.146:0/1250576574 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa7f80601e0 0x7fa7f8106520 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7fa7ec014000 tx=0x7fa7ec00b040).stop 2024-01-21T20:06:49.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:49.773+0000 7fa7f7fff700 1 -- 172.21.15.146:0/1250576574 shutdown_connections 2024-01-21T20:06:49.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:49.773+0000 7fa7f7fff700 1 --2- 172.21.15.146:0/1250576574 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fa7e002ff60 0x7fa7e0032420 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:49.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:49.773+0000 7fa7f7fff700 1 --2- 172.21.15.146:0/1250576574 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa7f80601e0 0x7fa7f8106520 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:49.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:49.773+0000 7fa7f7fff700 1 -- 172.21.15.146:0/1250576574 >> 172.21.15.146:0/1250576574 conn(0x7fa7f80f09a0 msgr2=0x7fa7f80f1450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:49.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:49.773+0000 7fa7f7fff700 1 -- 172.21.15.146:0/1250576574 shutdown_connections 2024-01-21T20:06:49.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:49.773+0000 7fa7f7fff700 1 -- 172.21.15.146:0/1250576574 wait complete. 2024-01-21T20:06:49.763 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:06:50.182 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:49 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:06:49.771578+0000 mon.smithi053 (mon.0) 291 : audit [DBG] from='client.? 172.21.15.146:0/1250576574' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:06:51.166 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:06:51.166 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:06:51.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:50 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:06:49.199188+0000 mgr.smithi053.fgsmol (mgr.14164) 238 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:52.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:52.418+0000 7f83acfd4700 1 -- 172.21.15.146:0/2436072968 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83a80f5410 msgr2=0x7f83a80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:52.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:52.418+0000 7f83acfd4700 1 --2- 172.21.15.146:0/2436072968 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83a80f5410 0x7f83a80f5830 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f8398009fb0 tx=0x7f8398009b30).stop 2024-01-21T20:06:52.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:52.418+0000 7f83acfd4700 1 -- 172.21.15.146:0/2436072968 shutdown_connections 2024-01-21T20:06:52.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:52.418+0000 7f83acfd4700 1 --2- 172.21.15.146:0/2436072968 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83a80f5410 0x7f83a80f5830 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:52.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:52.418+0000 7f83acfd4700 1 -- 172.21.15.146:0/2436072968 >> 172.21.15.146:0/2436072968 conn(0x7f83a80f0b70 msgr2=0x7f83a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:52.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:52.418+0000 7f83acfd4700 1 -- 172.21.15.146:0/2436072968 shutdown_connections 2024-01-21T20:06:52.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:52.418+0000 7f83acfd4700 1 -- 172.21.15.146:0/2436072968 wait complete. 2024-01-21T20:06:52.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:52.419+0000 7f83acfd4700 1 Processor -- start 2024-01-21T20:06:52.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:52.419+0000 7f83acfd4700 1 -- start start 2024-01-21T20:06:52.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:52.419+0000 7f83acfd4700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83a80f5410 0x7f83a8105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:52.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:52.419+0000 7f83acfd4700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a8105b20 con 0x7f83a80f5410 2024-01-21T20:06:52.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:52.419+0000 7f83a659c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83a80f5410 0x7f83a8105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:52.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:52.419+0000 7f83a659c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83a80f5410 0x7f83a8105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:42578/0 (socket says 172.21.15.146:42578) 2024-01-21T20:06:52.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:52.419+0000 7f83a659c700 1 -- 172.21.15.146:0/783917922 learned_addr learned my addr 172.21.15.146:0/783917922 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:06:52.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:52.420+0000 7f83a659c700 1 -- 172.21.15.146:0/783917922 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8398005040 con 0x7f83a80f5410 2024-01-21T20:06:52.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:52.420+0000 7f83a659c700 1 --2- 172.21.15.146:0/783917922 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83a80f5410 0x7f83a8105630 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f8398013510 tx=0x7f839800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:52.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:52.420+0000 7f8396ffd700 1 -- 172.21.15.146:0/783917922 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8398009320 con 0x7f83a80f5410 2024-01-21T20:06:52.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:52.420+0000 7f83acfd4700 1 -- 172.21.15.146:0/783917922 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83a8105d20 con 0x7f83a80f5410 2024-01-21T20:06:52.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:52.420+0000 7f8396ffd700 1 -- 172.21.15.146:0/783917922 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8398013f00 con 0x7f83a80f5410 2024-01-21T20:06:52.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:52.420+0000 7f8396ffd700 1 -- 172.21.15.146:0/783917922 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8398014d90 con 0x7f83a80f5410 2024-01-21T20:06:52.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:52.420+0000 7f83acfd4700 1 -- 172.21.15.146:0/783917922 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83a81061b0 con 0x7f83a80f5410 2024-01-21T20:06:52.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:52.421+0000 7f8396ffd700 1 -- 172.21.15.146:0/783917922 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f83980095f0 con 0x7f83a80f5410 2024-01-21T20:06:52.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:52.421+0000 7f83acfd4700 1 -- 172.21.15.146:0/783917922 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83a8045ad0 con 0x7f83a80f5410 2024-01-21T20:06:52.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:52.421+0000 7f8396ffd700 1 --2- 172.21.15.146:0/783917922 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f839002ffb0 0x7f8390032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:52.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:52.421+0000 7f8396ffd700 1 -- 172.21.15.146:0/783917922 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f83980410a0 con 0x7f83a80f5410 2024-01-21T20:06:52.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:52.421+0000 7f83a5d9b700 1 --2- 172.21.15.146:0/783917922 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f839002ffb0 0x7f8390032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:52.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:52.422+0000 7f83a5d9b700 1 --2- 172.21.15.146:0/783917922 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f839002ffb0 0x7f8390032470 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f839c0096a0 tx=0x7f839c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:52.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:52.426+0000 7f8396ffd700 1 -- 172.21.15.146:0/783917922 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f839806d2b0 con 0x7f83a80f5410 2024-01-21T20:06:52.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:52.634+0000 7f83acfd4700 1 -- 172.21.15.146:0/783917922 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f83a80fae60 con 0x7f83a80f5410 2024-01-21T20:06:52.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:52.634+0000 7f8396ffd700 1 -- 172.21.15.146:0/783917922 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8398014420 con 0x7f83a80f5410 2024-01-21T20:06:52.624 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:06:52.624 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:06:52.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:52.636+0000 7f83acfd4700 1 -- 172.21.15.146:0/783917922 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f839002ffb0 msgr2=0x7f8390032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:52.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:52.636+0000 7f83acfd4700 1 --2- 172.21.15.146:0/783917922 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f839002ffb0 0x7f8390032470 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f839c0096a0 tx=0x7f839c006b40).stop 2024-01-21T20:06:52.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:52.636+0000 7f83acfd4700 1 -- 172.21.15.146:0/783917922 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83a80f5410 msgr2=0x7f83a8105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:52.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:52.636+0000 7f83acfd4700 1 --2- 172.21.15.146:0/783917922 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83a80f5410 0x7f83a8105630 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f8398013510 tx=0x7f839800b040).stop 2024-01-21T20:06:52.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:52.636+0000 7f83acfd4700 1 -- 172.21.15.146:0/783917922 shutdown_connections 2024-01-21T20:06:52.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:52.636+0000 7f83acfd4700 1 --2- 172.21.15.146:0/783917922 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f839002ffb0 0x7f8390032470 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:52.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:52.636+0000 7f83acfd4700 1 --2- 172.21.15.146:0/783917922 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83a80f5410 0x7f83a8105630 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:52.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:52.636+0000 7f83acfd4700 1 -- 172.21.15.146:0/783917922 >> 172.21.15.146:0/783917922 conn(0x7f83a80f0b70 msgr2=0x7f83a80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:52.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:52.636+0000 7f83acfd4700 1 -- 172.21.15.146:0/783917922 shutdown_connections 2024-01-21T20:06:52.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:52.636+0000 7f83acfd4700 1 -- 172.21.15.146:0/783917922 wait complete. 2024-01-21T20:06:52.627 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:06:53.433 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:52 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:06:51.199457+0000 mgr.smithi053.fgsmol (mgr.14164) 239 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:53.433 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:52 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:06:52.634776+0000 mon.smithi053 (mon.0) 292 : audit [DBG] from='client.? 172.21.15.146:0/783917922' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:06:54.058 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:06:54.059 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:06:55.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:55.308+0000 7f80279b8700 1 -- 172.21.15.146:0/1556048905 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80200f6510 msgr2=0x7f80200f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:55.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:55.308+0000 7f80279b8700 1 --2- 172.21.15.146:0/1556048905 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80200f6510 0x7f80200f6930 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f8010009fb0 tx=0x7f8010009b30).stop 2024-01-21T20:06:55.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:55.308+0000 7f80279b8700 1 -- 172.21.15.146:0/1556048905 shutdown_connections 2024-01-21T20:06:55.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:55.308+0000 7f80279b8700 1 --2- 172.21.15.146:0/1556048905 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80200f6510 0x7f80200f6930 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:55.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:55.308+0000 7f80279b8700 1 -- 172.21.15.146:0/1556048905 >> 172.21.15.146:0/1556048905 conn(0x7f80200f1c70 msgr2=0x7f80200f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:55.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:55.308+0000 7f80279b8700 1 -- 172.21.15.146:0/1556048905 shutdown_connections 2024-01-21T20:06:55.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:55.308+0000 7f80279b8700 1 -- 172.21.15.146:0/1556048905 wait complete. 2024-01-21T20:06:55.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:55.309+0000 7f80279b8700 1 Processor -- start 2024-01-21T20:06:55.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:55.309+0000 7f80279b8700 1 -- start start 2024-01-21T20:06:55.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:55.309+0000 7f80279b8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80200f6510 0x7f8020106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:55.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:55.309+0000 7f80279b8700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8020106c20 con 0x7f80200f6510 2024-01-21T20:06:55.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:55.309+0000 7f8025754700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80200f6510 0x7f8020106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:55.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:55.309+0000 7f8025754700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80200f6510 0x7f8020106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:42602/0 (socket says 172.21.15.146:42602) 2024-01-21T20:06:55.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:55.309+0000 7f8025754700 1 -- 172.21.15.146:0/1889085143 learned_addr learned my addr 172.21.15.146:0/1889085143 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:06:55.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:55.310+0000 7f8025754700 1 -- 172.21.15.146:0/1889085143 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8010005040 con 0x7f80200f6510 2024-01-21T20:06:55.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:55.310+0000 7f8025754700 1 --2- 172.21.15.146:0/1889085143 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80200f6510 0x7f8020106730 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f8010013510 tx=0x7f801000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:55.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:55.310+0000 7f8015ffb700 1 -- 172.21.15.146:0/1889085143 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8010009320 con 0x7f80200f6510 2024-01-21T20:06:55.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:55.310+0000 7f80279b8700 1 -- 172.21.15.146:0/1889085143 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8020106e20 con 0x7f80200f6510 2024-01-21T20:06:55.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:55.310+0000 7f8015ffb700 1 -- 172.21.15.146:0/1889085143 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8010013f00 con 0x7f80200f6510 2024-01-21T20:06:55.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:55.310+0000 7f8015ffb700 1 -- 172.21.15.146:0/1889085143 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8010014d90 con 0x7f80200f6510 2024-01-21T20:06:55.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:55.310+0000 7f80279b8700 1 -- 172.21.15.146:0/1889085143 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80201072b0 con 0x7f80200f6510 2024-01-21T20:06:55.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:55.311+0000 7f80279b8700 1 -- 172.21.15.146:0/1889085143 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8020045ad0 con 0x7f80200f6510 2024-01-21T20:06:55.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:55.311+0000 7f8015ffb700 1 -- 172.21.15.146:0/1889085143 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f80100095f0 con 0x7f80200f6510 2024-01-21T20:06:55.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:55.311+0000 7f8015ffb700 1 --2- 172.21.15.146:0/1889085143 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f800c02ff60 0x7f800c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:55.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:55.311+0000 7f8015ffb700 1 -- 172.21.15.146:0/1889085143 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8010041080 con 0x7f80200f6510 2024-01-21T20:06:55.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:55.311+0000 7f8024f53700 1 --2- 172.21.15.146:0/1889085143 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f800c02ff60 0x7f800c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:55.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:55.312+0000 7f8024f53700 1 --2- 172.21.15.146:0/1889085143 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f800c02ff60 0x7f800c032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f801c0096a0 tx=0x7f801c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:55.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:55.316+0000 7f8015ffb700 1 -- 172.21.15.146:0/1889085143 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8010014420 con 0x7f80200f6510 2024-01-21T20:06:55.433 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:54 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:06:53.199913+0000 mgr.smithi053.fgsmol (mgr.14164) 240 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:55.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:55.522+0000 7f80279b8700 1 -- 172.21.15.146:0/1889085143 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f80200fbf60 con 0x7f80200f6510 2024-01-21T20:06:55.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:55.523+0000 7f8015ffb700 1 -- 172.21.15.146:0/1889085143 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8010010230 con 0x7f80200f6510 2024-01-21T20:06:55.514 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:06:55.514 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:06:55.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:55.524+0000 7f80279b8700 1 -- 172.21.15.146:0/1889085143 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f800c02ff60 msgr2=0x7f800c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:55.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:55.524+0000 7f80279b8700 1 --2- 172.21.15.146:0/1889085143 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f800c02ff60 0x7f800c032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f801c0096a0 tx=0x7f801c006b40).stop 2024-01-21T20:06:55.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:55.524+0000 7f80279b8700 1 -- 172.21.15.146:0/1889085143 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80200f6510 msgr2=0x7f8020106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:55.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:55.524+0000 7f80279b8700 1 --2- 172.21.15.146:0/1889085143 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80200f6510 0x7f8020106730 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f8010013510 tx=0x7f801000b040).stop 2024-01-21T20:06:55.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:55.524+0000 7f80279b8700 1 -- 172.21.15.146:0/1889085143 shutdown_connections 2024-01-21T20:06:55.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:55.524+0000 7f80279b8700 1 --2- 172.21.15.146:0/1889085143 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f800c02ff60 0x7f800c032420 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:55.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:55.524+0000 7f80279b8700 1 --2- 172.21.15.146:0/1889085143 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80200f6510 0x7f8020106730 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:55.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:55.524+0000 7f80279b8700 1 -- 172.21.15.146:0/1889085143 >> 172.21.15.146:0/1889085143 conn(0x7f80200f1c70 msgr2=0x7f80200f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:55.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:55.524+0000 7f80279b8700 1 -- 172.21.15.146:0/1889085143 shutdown_connections 2024-01-21T20:06:55.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:55.525+0000 7f80279b8700 1 -- 172.21.15.146:0/1889085143 wait complete. 2024-01-21T20:06:55.516 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:06:56.433 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:55 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:06:55.524364+0000 mon.smithi053 (mon.0) 293 : audit [DBG] from='client.? 172.21.15.146:0/1889085143' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:06:56.952 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:06:56.952 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:06:57.433 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:56 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:06:55.200329+0000 mgr.smithi053.fgsmol (mgr.14164) 241 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:58.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:58.210+0000 7fbea2f80700 1 -- 172.21.15.146:0/2910013660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe9c0f6510 msgr2=0x7fbe9c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:58.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:58.210+0000 7fbea2f80700 1 --2- 172.21.15.146:0/2910013660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe9c0f6510 0x7fbe9c0f6930 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7fbe90009fb0 tx=0x7fbe90009b30).stop 2024-01-21T20:06:58.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:58.210+0000 7fbea2f80700 1 -- 172.21.15.146:0/2910013660 shutdown_connections 2024-01-21T20:06:58.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:58.210+0000 7fbea2f80700 1 --2- 172.21.15.146:0/2910013660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe9c0f6510 0x7fbe9c0f6930 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:58.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:58.210+0000 7fbea2f80700 1 -- 172.21.15.146:0/2910013660 >> 172.21.15.146:0/2910013660 conn(0x7fbe9c0f1c70 msgr2=0x7fbe9c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:58.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:58.210+0000 7fbea2f80700 1 -- 172.21.15.146:0/2910013660 shutdown_connections 2024-01-21T20:06:58.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:58.210+0000 7fbea2f80700 1 -- 172.21.15.146:0/2910013660 wait complete. 2024-01-21T20:06:58.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:58.211+0000 7fbea2f80700 1 Processor -- start 2024-01-21T20:06:58.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:58.211+0000 7fbea2f80700 1 -- start start 2024-01-21T20:06:58.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:58.211+0000 7fbea2f80700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe9c0f6510 0x7fbe9c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:58.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:58.211+0000 7fbea2f80700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c106c20 con 0x7fbe9c0f6510 2024-01-21T20:06:58.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:58.211+0000 7fbea0d1c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe9c0f6510 0x7fbe9c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:58.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:58.211+0000 7fbea0d1c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe9c0f6510 0x7fbe9c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:52904/0 (socket says 172.21.15.146:52904) 2024-01-21T20:06:58.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:58.211+0000 7fbea0d1c700 1 -- 172.21.15.146:0/1861891130 learned_addr learned my addr 172.21.15.146:0/1861891130 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:06:58.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:58.212+0000 7fbea0d1c700 1 -- 172.21.15.146:0/1861891130 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe90005040 con 0x7fbe9c0f6510 2024-01-21T20:06:58.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:58.212+0000 7fbea0d1c700 1 --2- 172.21.15.146:0/1861891130 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe9c0f6510 0x7fbe9c106730 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7fbe90013510 tx=0x7fbe9000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:58.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:58.212+0000 7fbe997fa700 1 -- 172.21.15.146:0/1861891130 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe90009320 con 0x7fbe9c0f6510 2024-01-21T20:06:58.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:58.212+0000 7fbea2f80700 1 -- 172.21.15.146:0/1861891130 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe9c106e20 con 0x7fbe9c0f6510 2024-01-21T20:06:58.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:58.212+0000 7fbe997fa700 1 -- 172.21.15.146:0/1861891130 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbe90013f00 con 0x7fbe9c0f6510 2024-01-21T20:06:58.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:58.212+0000 7fbe997fa700 1 -- 172.21.15.146:0/1861891130 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe90014d90 con 0x7fbe9c0f6510 2024-01-21T20:06:58.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:58.212+0000 7fbea2f80700 1 -- 172.21.15.146:0/1861891130 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe9c1072b0 con 0x7fbe9c0f6510 2024-01-21T20:06:58.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:58.213+0000 7fbe997fa700 1 -- 172.21.15.146:0/1861891130 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbe90014420 con 0x7fbe9c0f6510 2024-01-21T20:06:58.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:58.213+0000 7fbea2f80700 1 -- 172.21.15.146:0/1861891130 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe9c045ad0 con 0x7fbe9c0f6510 2024-01-21T20:06:58.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:58.213+0000 7fbe997fa700 1 --2- 172.21.15.146:0/1861891130 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fbe8c02ff60 0x7fbe8c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:58.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:58.213+0000 7fbe997fa700 1 -- 172.21.15.146:0/1861891130 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbe90041000 con 0x7fbe9c0f6510 2024-01-21T20:06:58.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:58.213+0000 7fbe9bfff700 1 --2- 172.21.15.146:0/1861891130 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fbe8c02ff60 0x7fbe8c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:58.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:58.214+0000 7fbe9bfff700 1 --2- 172.21.15.146:0/1861891130 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fbe8c02ff60 0x7fbe8c032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fbe880096a0 tx=0x7fbe88006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:58.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:58.218+0000 7fbe997fa700 1 -- 172.21.15.146:0/1861891130 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbe900095f0 con 0x7fbe9c0f6510 2024-01-21T20:06:58.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:58.421+0000 7fbea2f80700 1 -- 172.21.15.146:0/1861891130 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbe9c0fbf60 con 0x7fbe9c0f6510 2024-01-21T20:06:58.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:58.422+0000 7fbe997fa700 1 -- 172.21.15.146:0/1861891130 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbe9001db70 con 0x7fbe9c0f6510 2024-01-21T20:06:58.413 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:06:58.413 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:06:58.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:58.423+0000 7fbea2f80700 1 -- 172.21.15.146:0/1861891130 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fbe8c02ff60 msgr2=0x7fbe8c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:58.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:58.423+0000 7fbea2f80700 1 --2- 172.21.15.146:0/1861891130 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fbe8c02ff60 0x7fbe8c032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fbe880096a0 tx=0x7fbe88006b40).stop 2024-01-21T20:06:58.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:58.423+0000 7fbea2f80700 1 -- 172.21.15.146:0/1861891130 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe9c0f6510 msgr2=0x7fbe9c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:58.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:58.423+0000 7fbea2f80700 1 --2- 172.21.15.146:0/1861891130 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe9c0f6510 0x7fbe9c106730 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7fbe90013510 tx=0x7fbe9000b040).stop 2024-01-21T20:06:58.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:58.424+0000 7fbea2f80700 1 -- 172.21.15.146:0/1861891130 shutdown_connections 2024-01-21T20:06:58.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:58.424+0000 7fbea2f80700 1 --2- 172.21.15.146:0/1861891130 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fbe8c02ff60 0x7fbe8c032420 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:58.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:58.424+0000 7fbea2f80700 1 --2- 172.21.15.146:0/1861891130 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe9c0f6510 0x7fbe9c106730 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:58.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:58.424+0000 7fbea2f80700 1 -- 172.21.15.146:0/1861891130 >> 172.21.15.146:0/1861891130 conn(0x7fbe9c0f1c70 msgr2=0x7fbe9c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:58.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:58.424+0000 7fbea2f80700 1 -- 172.21.15.146:0/1861891130 shutdown_connections 2024-01-21T20:06:58.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:06:58.424+0000 7fbea2f80700 1 -- 172.21.15.146:0/1861891130 wait complete. 2024-01-21T20:06:58.415 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:06:59.432 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:58 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:06:57.200657+0000 mgr.smithi053.fgsmol (mgr.14164) 242 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:59.433 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:06:58 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:06:58.422986+0000 mon.smithi053 (mon.0) 294 : audit [DBG] from='client.? 172.21.15.146:0/1861891130' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:06:59.811 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:06:59.812 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:07:01.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:01.055+0000 7ff661b4a700 1 -- 172.21.15.146:0/1836411346 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff65c0f51a0 msgr2=0x7ff65c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:01.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:01.055+0000 7ff661b4a700 1 --2- 172.21.15.146:0/1836411346 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff65c0f51a0 0x7ff65c0f55c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7ff650009fb0 tx=0x7ff650009b30).stop 2024-01-21T20:07:01.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:01.056+0000 7ff661b4a700 1 -- 172.21.15.146:0/1836411346 shutdown_connections 2024-01-21T20:07:01.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:01.056+0000 7ff661b4a700 1 --2- 172.21.15.146:0/1836411346 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff65c0f51a0 0x7ff65c0f55c0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:01.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:01.056+0000 7ff661b4a700 1 -- 172.21.15.146:0/1836411346 >> 172.21.15.146:0/1836411346 conn(0x7ff65c0f09a0 msgr2=0x7ff65c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:01.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:01.056+0000 7ff661b4a700 1 -- 172.21.15.146:0/1836411346 shutdown_connections 2024-01-21T20:07:01.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:01.056+0000 7ff661b4a700 1 -- 172.21.15.146:0/1836411346 wait complete. 2024-01-21T20:07:01.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:01.056+0000 7ff661b4a700 1 Processor -- start 2024-01-21T20:07:01.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:01.056+0000 7ff661b4a700 1 -- start start 2024-01-21T20:07:01.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:01.056+0000 7ff661b4a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff65c0f51a0 0x7ff65c1064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:01.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:01.056+0000 7ff661b4a700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff65c106990 con 0x7ff65c0f51a0 2024-01-21T20:07:01.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:01.057+0000 7ff660b48700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff65c0f51a0 0x7ff65c1064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:01.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:01.057+0000 7ff660b48700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff65c0f51a0 0x7ff65c1064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:52928/0 (socket says 172.21.15.146:52928) 2024-01-21T20:07:01.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:01.057+0000 7ff660b48700 1 -- 172.21.15.146:0/426211365 learned_addr learned my addr 172.21.15.146:0/426211365 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:07:01.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:01.057+0000 7ff660b48700 1 -- 172.21.15.146:0/426211365 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff650005040 con 0x7ff65c0f51a0 2024-01-21T20:07:01.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:01.057+0000 7ff660b48700 1 --2- 172.21.15.146:0/426211365 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff65c0f51a0 0x7ff65c1064a0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7ff650013510 tx=0x7ff65000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:01.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:01.057+0000 7ff6597fa700 1 -- 172.21.15.146:0/426211365 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff650009320 con 0x7ff65c0f51a0 2024-01-21T20:07:01.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:01.057+0000 7ff661b4a700 1 -- 172.21.15.146:0/426211365 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff65c106b90 con 0x7ff65c0f51a0 2024-01-21T20:07:01.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:01.057+0000 7ff6597fa700 1 -- 172.21.15.146:0/426211365 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff650013f00 con 0x7ff65c0f51a0 2024-01-21T20:07:01.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:01.057+0000 7ff6597fa700 1 -- 172.21.15.146:0/426211365 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff650014dc0 con 0x7ff65c0f51a0 2024-01-21T20:07:01.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:01.057+0000 7ff661b4a700 1 -- 172.21.15.146:0/426211365 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff65c107020 con 0x7ff65c0f51a0 2024-01-21T20:07:01.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:01.058+0000 7ff6597fa700 1 -- 172.21.15.146:0/426211365 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff6500095f0 con 0x7ff65c0f51a0 2024-01-21T20:07:01.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:01.058+0000 7ff661b4a700 1 -- 172.21.15.146:0/426211365 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff65c045ad0 con 0x7ff65c0f51a0 2024-01-21T20:07:01.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:01.058+0000 7ff6597fa700 1 --2- 172.21.15.146:0/426211365 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff64c02ff60 0x7ff64c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:01.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:01.058+0000 7ff6597fa700 1 -- 172.21.15.146:0/426211365 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff650041100 con 0x7ff65c0f51a0 2024-01-21T20:07:01.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:01.058+0000 7ff65bfff700 1 --2- 172.21.15.146:0/426211365 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff64c02ff60 0x7ff64c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:01.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:01.059+0000 7ff65bfff700 1 --2- 172.21.15.146:0/426211365 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff64c02ff60 0x7ff64c032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7ff6480096a0 tx=0x7ff648006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:01.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:01.063+0000 7ff6597fa700 1 -- 172.21.15.146:0/426211365 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff650014420 con 0x7ff65c0f51a0 2024-01-21T20:07:01.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:01.268+0000 7ff661b4a700 1 -- 172.21.15.146:0/426211365 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff65c107ed0 con 0x7ff65c0f51a0 2024-01-21T20:07:01.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:01.269+0000 7ff6597fa700 1 -- 172.21.15.146:0/426211365 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff650010230 con 0x7ff65c0f51a0 2024-01-21T20:07:01.260 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:07:01.260 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:07:01.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:01.271+0000 7ff661b4a700 1 -- 172.21.15.146:0/426211365 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff64c02ff60 msgr2=0x7ff64c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:01.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:01.271+0000 7ff661b4a700 1 --2- 172.21.15.146:0/426211365 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff64c02ff60 0x7ff64c032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7ff6480096a0 tx=0x7ff648006b40).stop 2024-01-21T20:07:01.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:01.271+0000 7ff661b4a700 1 -- 172.21.15.146:0/426211365 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff65c0f51a0 msgr2=0x7ff65c1064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:01.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:01.271+0000 7ff661b4a700 1 --2- 172.21.15.146:0/426211365 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff65c0f51a0 0x7ff65c1064a0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7ff650013510 tx=0x7ff65000b040).stop 2024-01-21T20:07:01.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:01.271+0000 7ff661b4a700 1 -- 172.21.15.146:0/426211365 shutdown_connections 2024-01-21T20:07:01.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:01.271+0000 7ff661b4a700 1 --2- 172.21.15.146:0/426211365 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff64c02ff60 0x7ff64c032420 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:01.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:01.271+0000 7ff661b4a700 1 --2- 172.21.15.146:0/426211365 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff65c0f51a0 0x7ff65c1064a0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:01.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:01.271+0000 7ff661b4a700 1 -- 172.21.15.146:0/426211365 >> 172.21.15.146:0/426211365 conn(0x7ff65c0f09a0 msgr2=0x7ff65c05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:01.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:01.271+0000 7ff661b4a700 1 -- 172.21.15.146:0/426211365 shutdown_connections 2024-01-21T20:07:01.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:01.271+0000 7ff661b4a700 1 -- 172.21.15.146:0/426211365 wait complete. 2024-01-21T20:07:01.263 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:07:01.432 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:00 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:06:59.201070+0000 mgr.smithi053.fgsmol (mgr.14164) 243 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:02.433 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:01 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:07:01.270388+0000 mon.smithi053 (mon.0) 295 : audit [DBG] from='client.? 172.21.15.146:0/426211365' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:07:02.678 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:07:02.678 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:07:03.433 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:02 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:07:01.201424+0000 mgr.smithi053.fgsmol (mgr.14164) 244 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:03.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:03.923+0000 7f2b5ce49700 1 -- 172.21.15.146:0/825029643 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b580f51a0 msgr2=0x7f2b580f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:03.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:03.923+0000 7f2b5ce49700 1 --2- 172.21.15.146:0/825029643 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b580f51a0 0x7f2b580f55c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f2b48009fb0 tx=0x7f2b48009b30).stop 2024-01-21T20:07:03.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:03.923+0000 7f2b5ce49700 1 -- 172.21.15.146:0/825029643 shutdown_connections 2024-01-21T20:07:03.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:03.923+0000 7f2b5ce49700 1 --2- 172.21.15.146:0/825029643 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b580f51a0 0x7f2b580f55c0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:03.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:03.923+0000 7f2b5ce49700 1 -- 172.21.15.146:0/825029643 >> 172.21.15.146:0/825029643 conn(0x7f2b580f09a0 msgr2=0x7f2b580f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:03.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:03.923+0000 7f2b5ce49700 1 -- 172.21.15.146:0/825029643 shutdown_connections 2024-01-21T20:07:03.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:03.923+0000 7f2b5ce49700 1 -- 172.21.15.146:0/825029643 wait complete. 2024-01-21T20:07:03.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:03.923+0000 7f2b5ce49700 1 Processor -- start 2024-01-21T20:07:03.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:03.924+0000 7f2b5ce49700 1 -- start start 2024-01-21T20:07:03.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:03.924+0000 7f2b5ce49700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b580f51a0 0x7f2b581053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:03.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:03.924+0000 7f2b5ce49700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b581058a0 con 0x7f2b580f51a0 2024-01-21T20:07:03.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:03.924+0000 7f2b577fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b580f51a0 0x7f2b581053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:03.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:03.924+0000 7f2b577fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b580f51a0 0x7f2b581053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:52950/0 (socket says 172.21.15.146:52950) 2024-01-21T20:07:03.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:03.924+0000 7f2b577fe700 1 -- 172.21.15.146:0/1370613843 learned_addr learned my addr 172.21.15.146:0/1370613843 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:07:03.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:03.925+0000 7f2b577fe700 1 -- 172.21.15.146:0/1370613843 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b48005040 con 0x7f2b580f51a0 2024-01-21T20:07:03.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:03.925+0000 7f2b577fe700 1 --2- 172.21.15.146:0/1370613843 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b580f51a0 0x7f2b581053b0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f2b48009830 tx=0x7f2b4800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:03.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:03.925+0000 7f2b3ffff700 1 -- 172.21.15.146:0/1370613843 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b48009320 con 0x7f2b580f51a0 2024-01-21T20:07:03.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:03.925+0000 7f2b5ce49700 1 -- 172.21.15.146:0/1370613843 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b58105aa0 con 0x7f2b580f51a0 2024-01-21T20:07:03.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:03.925+0000 7f2b3ffff700 1 -- 172.21.15.146:0/1370613843 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2b480136c0 con 0x7f2b580f51a0 2024-01-21T20:07:03.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:03.925+0000 7f2b3ffff700 1 -- 172.21.15.146:0/1370613843 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b4801ee90 con 0x7f2b580f51a0 2024-01-21T20:07:03.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:03.925+0000 7f2b5ce49700 1 -- 172.21.15.146:0/1370613843 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b58105f30 con 0x7f2b580f51a0 2024-01-21T20:07:03.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:03.926+0000 7f2b5ce49700 1 -- 172.21.15.146:0/1370613843 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b58045ad0 con 0x7f2b580f51a0 2024-01-21T20:07:03.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:03.926+0000 7f2b3ffff700 1 -- 172.21.15.146:0/1370613843 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2b4801e420 con 0x7f2b580f51a0 2024-01-21T20:07:03.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:03.926+0000 7f2b3ffff700 1 --2- 172.21.15.146:0/1370613843 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f2b4002ff10 0x7f2b400323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:03.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:03.926+0000 7f2b3ffff700 1 -- 172.21.15.146:0/1370613843 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2b48041010 con 0x7f2b580f51a0 2024-01-21T20:07:03.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:03.926+0000 7f2b56ffd700 1 --2- 172.21.15.146:0/1370613843 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f2b4002ff10 0x7f2b400323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:03.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:03.927+0000 7f2b56ffd700 1 --2- 172.21.15.146:0/1370613843 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f2b4002ff10 0x7f2b400323d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f2b4c0096a0 tx=0x7f2b4c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:03.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:03.931+0000 7f2b3ffff700 1 -- 172.21.15.146:0/1370613843 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2b480095f0 con 0x7f2b580f51a0 2024-01-21T20:07:04.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:04.135+0000 7f2b5ce49700 1 -- 172.21.15.146:0/1370613843 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2b58106de0 con 0x7f2b580f51a0 2024-01-21T20:07:04.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:04.135+0000 7f2b3ffff700 1 -- 172.21.15.146:0/1370613843 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2b4801cb10 con 0x7f2b580f51a0 2024-01-21T20:07:04.128 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:07:04.128 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:07:04.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:04.137+0000 7f2b5ce49700 1 -- 172.21.15.146:0/1370613843 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f2b4002ff10 msgr2=0x7f2b400323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:04.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:04.137+0000 7f2b5ce49700 1 --2- 172.21.15.146:0/1370613843 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f2b4002ff10 0x7f2b400323d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f2b4c0096a0 tx=0x7f2b4c006b40).stop 2024-01-21T20:07:04.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:04.137+0000 7f2b5ce49700 1 -- 172.21.15.146:0/1370613843 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b580f51a0 msgr2=0x7f2b581053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:04.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:04.137+0000 7f2b5ce49700 1 --2- 172.21.15.146:0/1370613843 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b580f51a0 0x7f2b581053b0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f2b48009830 tx=0x7f2b4800b040).stop 2024-01-21T20:07:04.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:04.137+0000 7f2b5ce49700 1 -- 172.21.15.146:0/1370613843 shutdown_connections 2024-01-21T20:07:04.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:04.137+0000 7f2b5ce49700 1 --2- 172.21.15.146:0/1370613843 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f2b4002ff10 0x7f2b400323d0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:04.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:04.137+0000 7f2b5ce49700 1 --2- 172.21.15.146:0/1370613843 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b580f51a0 0x7f2b581053b0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:04.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:04.137+0000 7f2b5ce49700 1 -- 172.21.15.146:0/1370613843 >> 172.21.15.146:0/1370613843 conn(0x7f2b580f09a0 msgr2=0x7f2b580f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:04.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:04.137+0000 7f2b5ce49700 1 -- 172.21.15.146:0/1370613843 shutdown_connections 2024-01-21T20:07:04.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:04.137+0000 7f2b5ce49700 1 -- 172.21.15.146:0/1370613843 wait complete. 2024-01-21T20:07:04.129 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:07:05.433 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:04 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:07:03.201632+0000 mgr.smithi053.fgsmol (mgr.14164) 245 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:05.433 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:04 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:07:04.136717+0000 mon.smithi053 (mon.0) 296 : audit [DBG] from='client.? 172.21.15.146:0/1370613843' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:07:05.565 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:07:05.616 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:07:06.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:06.824+0000 7fccd900c700 1 -- 172.21.15.146:0/561499972 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fccd40f5410 msgr2=0x7fccd40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:06.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:06.824+0000 7fccd900c700 1 --2- 172.21.15.146:0/561499972 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fccd40f5410 0x7fccd40f5830 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7fccc8009fb0 tx=0x7fccc8009b30).stop 2024-01-21T20:07:06.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:06.824+0000 7fccd900c700 1 -- 172.21.15.146:0/561499972 shutdown_connections 2024-01-21T20:07:06.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:06.824+0000 7fccd900c700 1 --2- 172.21.15.146:0/561499972 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fccd40f5410 0x7fccd40f5830 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:06.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:06.824+0000 7fccd900c700 1 -- 172.21.15.146:0/561499972 >> 172.21.15.146:0/561499972 conn(0x7fccd40f0b70 msgr2=0x7fccd40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:06.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:06.824+0000 7fccd900c700 1 -- 172.21.15.146:0/561499972 shutdown_connections 2024-01-21T20:07:06.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:06.824+0000 7fccd900c700 1 -- 172.21.15.146:0/561499972 wait complete. 2024-01-21T20:07:06.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:06.825+0000 7fccd900c700 1 Processor -- start 2024-01-21T20:07:06.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:06.825+0000 7fccd900c700 1 -- start start 2024-01-21T20:07:06.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:06.825+0000 7fccd900c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fccd40f5410 0x7fccd4109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:06.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:06.825+0000 7fccd900c700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd4060240 con 0x7fccd40f5410 2024-01-21T20:07:06.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:06.825+0000 7fccd2d9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fccd40f5410 0x7fccd4109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:06.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:06.825+0000 7fccd2d9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fccd40f5410 0x7fccd4109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:52966/0 (socket says 172.21.15.146:52966) 2024-01-21T20:07:06.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:06.825+0000 7fccd2d9d700 1 -- 172.21.15.146:0/4270799892 learned_addr learned my addr 172.21.15.146:0/4270799892 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:07:06.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:06.826+0000 7fccd2d9d700 1 -- 172.21.15.146:0/4270799892 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc8005040 con 0x7fccd40f5410 2024-01-21T20:07:06.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:06.826+0000 7fccd2d9d700 1 --2- 172.21.15.146:0/4270799892 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fccd40f5410 0x7fccd4109430 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7fccc8013510 tx=0x7fccc800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:06.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:06.826+0000 7fccbb7fe700 1 -- 172.21.15.146:0/4270799892 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fccc8009320 con 0x7fccd40f5410 2024-01-21T20:07:06.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:06.826+0000 7fccbb7fe700 1 -- 172.21.15.146:0/4270799892 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fccc8013f00 con 0x7fccd40f5410 2024-01-21T20:07:06.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:06.826+0000 7fccd900c700 1 -- 172.21.15.146:0/4270799892 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccd4060440 con 0x7fccd40f5410 2024-01-21T20:07:06.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:06.826+0000 7fccbb7fe700 1 -- 172.21.15.146:0/4270799892 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fccc8014d90 con 0x7fccd40f5410 2024-01-21T20:07:06.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:06.826+0000 7fccd900c700 1 -- 172.21.15.146:0/4270799892 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccd4109ae0 con 0x7fccd40f5410 2024-01-21T20:07:06.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:06.827+0000 7fccbb7fe700 1 -- 172.21.15.146:0/4270799892 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fccc80263a0 con 0x7fccd40f5410 2024-01-21T20:07:06.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:06.827+0000 7fccbb7fe700 1 --2- 172.21.15.146:0/4270799892 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fccbc02ff60 0x7fccbc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:06.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:06.827+0000 7fccbb7fe700 1 -- 172.21.15.146:0/4270799892 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fccc8040ff0 con 0x7fccd40f5410 2024-01-21T20:07:06.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:06.827+0000 7fccd900c700 1 -- 172.21.15.146:0/4270799892 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccd4045ad0 con 0x7fccd40f5410 2024-01-21T20:07:06.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:06.827+0000 7fccd259c700 1 --2- 172.21.15.146:0/4270799892 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fccbc02ff60 0x7fccbc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:06.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:06.828+0000 7fccd259c700 1 --2- 172.21.15.146:0/4270799892 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fccbc02ff60 0x7fccbc032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fccc40096a0 tx=0x7fccc4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:06.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:06.832+0000 7fccbb7fe700 1 -- 172.21.15.146:0/4270799892 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fccc8012920 con 0x7fccd40f5410 2024-01-21T20:07:07.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:07.039+0000 7fccd900c700 1 -- 172.21.15.146:0/4270799892 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fccd40faf50 con 0x7fccd40f5410 2024-01-21T20:07:07.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:07.039+0000 7fccbb7fe700 1 -- 172.21.15.146:0/4270799892 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fccc802b020 con 0x7fccd40f5410 2024-01-21T20:07:07.030 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:07:07.030 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:07:07.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:07.041+0000 7fccd900c700 1 -- 172.21.15.146:0/4270799892 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fccbc02ff60 msgr2=0x7fccbc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:07.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:07.041+0000 7fccd900c700 1 --2- 172.21.15.146:0/4270799892 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fccbc02ff60 0x7fccbc032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fccc40096a0 tx=0x7fccc4006b40).stop 2024-01-21T20:07:07.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:07.041+0000 7fccd900c700 1 -- 172.21.15.146:0/4270799892 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fccd40f5410 msgr2=0x7fccd4109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:07.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:07.041+0000 7fccd900c700 1 --2- 172.21.15.146:0/4270799892 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fccd40f5410 0x7fccd4109430 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7fccc8013510 tx=0x7fccc800b040).stop 2024-01-21T20:07:07.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:07.041+0000 7fccd900c700 1 -- 172.21.15.146:0/4270799892 shutdown_connections 2024-01-21T20:07:07.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:07.041+0000 7fccd900c700 1 --2- 172.21.15.146:0/4270799892 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fccbc02ff60 0x7fccbc032420 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:07.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:07.041+0000 7fccd900c700 1 --2- 172.21.15.146:0/4270799892 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fccd40f5410 0x7fccd4109430 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:07.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:07.041+0000 7fccd900c700 1 -- 172.21.15.146:0/4270799892 >> 172.21.15.146:0/4270799892 conn(0x7fccd40f0b70 msgr2=0x7fccd40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:07.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:07.041+0000 7fccd900c700 1 -- 172.21.15.146:0/4270799892 shutdown_connections 2024-01-21T20:07:07.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:07.041+0000 7fccd900c700 1 -- 172.21.15.146:0/4270799892 wait complete. 2024-01-21T20:07:07.032 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:07:07.432 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:06 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:07:05.201972+0000 mgr.smithi053.fgsmol (mgr.14164) 246 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:08.424 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:07:08.424 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:07:08.432 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:07 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:07:07.040701+0000 mon.smithi053 (mon.0) 297 : audit [DBG] from='client.? 172.21.15.146:0/4270799892' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:07:09.433 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:08 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:07:07.202312+0000 mgr.smithi053.fgsmol (mgr.14164) 247 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:09.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:09.662+0000 7f0e3e740700 1 -- 172.21.15.146:0/2638689678 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e38060280 msgr2=0x7f0e380f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:09.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:09.662+0000 7f0e3e740700 1 --2- 172.21.15.146:0/2638689678 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e38060280 0x7f0e380f90f0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f0e28009fb0 tx=0x7f0e28009b30).stop 2024-01-21T20:07:09.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:09.662+0000 7f0e3e740700 1 -- 172.21.15.146:0/2638689678 shutdown_connections 2024-01-21T20:07:09.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:09.662+0000 7f0e3e740700 1 --2- 172.21.15.146:0/2638689678 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e38060280 0x7f0e380f90f0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:09.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:09.662+0000 7f0e3e740700 1 -- 172.21.15.146:0/2638689678 >> 172.21.15.146:0/2638689678 conn(0x7f0e380f0b70 msgr2=0x7f0e380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:09.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:09.662+0000 7f0e3e740700 1 -- 172.21.15.146:0/2638689678 shutdown_connections 2024-01-21T20:07:09.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:09.662+0000 7f0e3e740700 1 -- 172.21.15.146:0/2638689678 wait complete. 2024-01-21T20:07:09.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:09.663+0000 7f0e3e740700 1 Processor -- start 2024-01-21T20:07:09.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:09.663+0000 7f0e3e740700 1 -- start start 2024-01-21T20:07:09.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:09.663+0000 7f0e3e740700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e38060280 0x7f0e381066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:09.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:09.663+0000 7f0e3e740700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e38106bd0 con 0x7f0e38060280 2024-01-21T20:07:09.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:09.663+0000 7f0e37fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e38060280 0x7f0e381066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:09.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:09.663+0000 7f0e37fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e38060280 0x7f0e381066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:59916/0 (socket says 172.21.15.146:59916) 2024-01-21T20:07:09.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:09.663+0000 7f0e37fff700 1 -- 172.21.15.146:0/4164964936 learned_addr learned my addr 172.21.15.146:0/4164964936 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:07:09.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:09.664+0000 7f0e37fff700 1 -- 172.21.15.146:0/4164964936 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e28005040 con 0x7f0e38060280 2024-01-21T20:07:09.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:09.664+0000 7f0e37fff700 1 --2- 172.21.15.146:0/4164964936 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e38060280 0x7f0e381066e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f0e28013510 tx=0x7f0e2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:09.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:09.664+0000 7f0e34ff9700 1 -- 172.21.15.146:0/4164964936 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e28009320 con 0x7f0e38060280 2024-01-21T20:07:09.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:09.664+0000 7f0e3e740700 1 -- 172.21.15.146:0/4164964936 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e38106dd0 con 0x7f0e38060280 2024-01-21T20:07:09.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:09.664+0000 7f0e34ff9700 1 -- 172.21.15.146:0/4164964936 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0e28013f00 con 0x7f0e38060280 2024-01-21T20:07:09.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:09.664+0000 7f0e34ff9700 1 -- 172.21.15.146:0/4164964936 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e28014dc0 con 0x7f0e38060280 2024-01-21T20:07:09.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:09.664+0000 7f0e3e740700 1 -- 172.21.15.146:0/4164964936 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e38107260 con 0x7f0e38060280 2024-01-21T20:07:09.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:09.665+0000 7f0e34ff9700 1 -- 172.21.15.146:0/4164964936 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0e280095f0 con 0x7f0e38060280 2024-01-21T20:07:09.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:09.665+0000 7f0e3e740700 1 -- 172.21.15.146:0/4164964936 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e38045ad0 con 0x7f0e38060280 2024-01-21T20:07:09.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:09.665+0000 7f0e34ff9700 1 --2- 172.21.15.146:0/4164964936 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0e2402ff60 0x7f0e24032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:09.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:09.665+0000 7f0e34ff9700 1 -- 172.21.15.146:0/4164964936 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0e28041000 con 0x7f0e38060280 2024-01-21T20:07:09.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:09.665+0000 7f0e377fe700 1 --2- 172.21.15.146:0/4164964936 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0e2402ff60 0x7f0e24032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:09.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:09.665+0000 7f0e377fe700 1 --2- 172.21.15.146:0/4164964936 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0e2402ff60 0x7f0e24032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f0e200096a0 tx=0x7f0e20006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:09.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:09.670+0000 7f0e34ff9700 1 -- 172.21.15.146:0/4164964936 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0e2801c790 con 0x7f0e38060280 2024-01-21T20:07:09.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:09.875+0000 7f0e3e740700 1 -- 172.21.15.146:0/4164964936 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0e38108110 con 0x7f0e38060280 2024-01-21T20:07:09.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:09.876+0000 7f0e34ff9700 1 -- 172.21.15.146:0/4164964936 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0e28010230 con 0x7f0e38060280 2024-01-21T20:07:09.867 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:07:09.867 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:07:09.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:09.877+0000 7f0e3e740700 1 -- 172.21.15.146:0/4164964936 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0e2402ff60 msgr2=0x7f0e24032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:09.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:09.877+0000 7f0e3e740700 1 --2- 172.21.15.146:0/4164964936 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0e2402ff60 0x7f0e24032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f0e200096a0 tx=0x7f0e20006b40).stop 2024-01-21T20:07:09.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:09.877+0000 7f0e3e740700 1 -- 172.21.15.146:0/4164964936 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e38060280 msgr2=0x7f0e381066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:09.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:09.877+0000 7f0e3e740700 1 --2- 172.21.15.146:0/4164964936 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e38060280 0x7f0e381066e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f0e28013510 tx=0x7f0e2800b040).stop 2024-01-21T20:07:09.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:09.877+0000 7f0e3e740700 1 -- 172.21.15.146:0/4164964936 shutdown_connections 2024-01-21T20:07:09.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:09.877+0000 7f0e3e740700 1 --2- 172.21.15.146:0/4164964936 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f0e2402ff60 0x7f0e24032420 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:09.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:09.877+0000 7f0e3e740700 1 --2- 172.21.15.146:0/4164964936 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e38060280 0x7f0e381066e0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:09.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:09.877+0000 7f0e3e740700 1 -- 172.21.15.146:0/4164964936 >> 172.21.15.146:0/4164964936 conn(0x7f0e380f0b70 msgr2=0x7f0e380f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:09.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:09.877+0000 7f0e3e740700 1 -- 172.21.15.146:0/4164964936 shutdown_connections 2024-01-21T20:07:09.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:09.877+0000 7f0e3e740700 1 -- 172.21.15.146:0/4164964936 wait complete. 2024-01-21T20:07:09.869 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:07:10.433 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:09 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:07:09.271449+0000 mon.smithi053 (mon.0) 298 : audit [INF] from='mgr.14164 172.21.15.53:0/2331671680' entity='mgr.smithi053.fgsmol' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fgsmol/mirror_snapshot_schedule"}]: dispatch 2024-01-21T20:07:10.433 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:09 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:07:09.279045+0000 mon.smithi053 (mon.0) 299 : audit [INF] from='mgr.14164 172.21.15.53:0/2331671680' entity='mgr.smithi053.fgsmol' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fgsmol/trash_purge_schedule"}]: dispatch 2024-01-21T20:07:10.433 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:09 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:07:09.877063+0000 mon.smithi053 (mon.0) 300 : audit [DBG] from='client.? 172.21.15.146:0/4164964936' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:07:11.280 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:07:11.280 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:07:11.433 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:10 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:07:09.202580+0000 mgr.smithi053.fgsmol (mgr.14164) 248 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:12.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:12.498+0000 7f083ca35700 1 -- 172.21.15.146:0/3545482715 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08380601e0 msgr2=0x7f08380f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:12.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:12.498+0000 7f083ca35700 1 --2- 172.21.15.146:0/3545482715 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08380601e0 0x7f08380f8f20 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f0828009fb0 tx=0x7f0828009b30).stop 2024-01-21T20:07:12.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:12.498+0000 7f083ca35700 1 -- 172.21.15.146:0/3545482715 shutdown_connections 2024-01-21T20:07:12.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:12.498+0000 7f083ca35700 1 --2- 172.21.15.146:0/3545482715 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08380601e0 0x7f08380f8f20 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:12.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:12.498+0000 7f083ca35700 1 -- 172.21.15.146:0/3545482715 >> 172.21.15.146:0/3545482715 conn(0x7f08380f09a0 msgr2=0x7f08380f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:12.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:12.498+0000 7f083ca35700 1 -- 172.21.15.146:0/3545482715 shutdown_connections 2024-01-21T20:07:12.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:12.498+0000 7f083ca35700 1 -- 172.21.15.146:0/3545482715 wait complete. 2024-01-21T20:07:12.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:12.499+0000 7f083ca35700 1 Processor -- start 2024-01-21T20:07:12.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:12.499+0000 7f083ca35700 1 -- start start 2024-01-21T20:07:12.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:12.499+0000 7f083ca35700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08380601e0 0x7f0838106520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:12.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:12.499+0000 7f083ca35700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0838106a10 con 0x7f08380601e0 2024-01-21T20:07:12.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:12.499+0000 7f0836d9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08380601e0 0x7f0838106520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:12.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:12.499+0000 7f0836d9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08380601e0 0x7f0838106520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:59934/0 (socket says 172.21.15.146:59934) 2024-01-21T20:07:12.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:12.499+0000 7f0836d9d700 1 -- 172.21.15.146:0/3588153620 learned_addr learned my addr 172.21.15.146:0/3588153620 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:07:12.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:12.500+0000 7f0836d9d700 1 -- 172.21.15.146:0/3588153620 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0828005040 con 0x7f08380601e0 2024-01-21T20:07:12.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:12.500+0000 7f0836d9d700 1 --2- 172.21.15.146:0/3588153620 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08380601e0 0x7f0838106520 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f0828013510 tx=0x7f082800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:12.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:12.500+0000 7f081f7fe700 1 -- 172.21.15.146:0/3588153620 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0828009320 con 0x7f08380601e0 2024-01-21T20:07:12.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:12.500+0000 7f083ca35700 1 -- 172.21.15.146:0/3588153620 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0838106c10 con 0x7f08380601e0 2024-01-21T20:07:12.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:12.500+0000 7f081f7fe700 1 -- 172.21.15.146:0/3588153620 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0828013f00 con 0x7f08380601e0 2024-01-21T20:07:12.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:12.500+0000 7f081f7fe700 1 -- 172.21.15.146:0/3588153620 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0828014d90 con 0x7f08380601e0 2024-01-21T20:07:12.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:12.500+0000 7f083ca35700 1 -- 172.21.15.146:0/3588153620 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08381070a0 con 0x7f08380601e0 2024-01-21T20:07:12.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:12.501+0000 7f081f7fe700 1 -- 172.21.15.146:0/3588153620 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0828014420 con 0x7f08380601e0 2024-01-21T20:07:12.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:12.501+0000 7f083ca35700 1 -- 172.21.15.146:0/3588153620 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0838045ad0 con 0x7f08380601e0 2024-01-21T20:07:12.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:12.501+0000 7f081f7fe700 1 --2- 172.21.15.146:0/3588153620 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f082002ff60 0x7f0820032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:12.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:12.501+0000 7f081f7fe700 1 -- 172.21.15.146:0/3588153620 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0828040ff0 con 0x7f08380601e0 2024-01-21T20:07:12.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:12.501+0000 7f083659c700 1 --2- 172.21.15.146:0/3588153620 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f082002ff60 0x7f0820032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:12.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:12.502+0000 7f083659c700 1 --2- 172.21.15.146:0/3588153620 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f082002ff60 0x7f0820032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f082c0096a0 tx=0x7f082c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:12.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:12.506+0000 7f081f7fe700 1 -- 172.21.15.146:0/3588153620 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f08280095f0 con 0x7f08380601e0 2024-01-21T20:07:12.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:12.708+0000 7f083ca35700 1 -- 172.21.15.146:0/3588153620 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0838107f50 con 0x7f08380601e0 2024-01-21T20:07:12.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:12.708+0000 7f081f7fe700 1 -- 172.21.15.146:0/3588153620 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f082801db70 con 0x7f08380601e0 2024-01-21T20:07:12.699 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:07:12.699 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:07:12.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:12.710+0000 7f083ca35700 1 -- 172.21.15.146:0/3588153620 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f082002ff60 msgr2=0x7f0820032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:12.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:12.710+0000 7f083ca35700 1 --2- 172.21.15.146:0/3588153620 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f082002ff60 0x7f0820032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f082c0096a0 tx=0x7f082c006b40).stop 2024-01-21T20:07:12.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:12.710+0000 7f083ca35700 1 -- 172.21.15.146:0/3588153620 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08380601e0 msgr2=0x7f0838106520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:12.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:12.710+0000 7f083ca35700 1 --2- 172.21.15.146:0/3588153620 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08380601e0 0x7f0838106520 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f0828013510 tx=0x7f082800b040).stop 2024-01-21T20:07:12.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:12.710+0000 7f083ca35700 1 -- 172.21.15.146:0/3588153620 shutdown_connections 2024-01-21T20:07:12.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:12.710+0000 7f083ca35700 1 --2- 172.21.15.146:0/3588153620 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f082002ff60 0x7f0820032420 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:12.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:12.710+0000 7f083ca35700 1 --2- 172.21.15.146:0/3588153620 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08380601e0 0x7f0838106520 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:12.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:12.710+0000 7f083ca35700 1 -- 172.21.15.146:0/3588153620 >> 172.21.15.146:0/3588153620 conn(0x7f08380f09a0 msgr2=0x7f08380f1450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:12.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:12.710+0000 7f083ca35700 1 -- 172.21.15.146:0/3588153620 shutdown_connections 2024-01-21T20:07:12.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:12.710+0000 7f083ca35700 1 -- 172.21.15.146:0/3588153620 wait complete. 2024-01-21T20:07:12.702 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:07:13.433 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:12 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:07:11.202833+0000 mgr.smithi053.fgsmol (mgr.14164) 249 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:13.433 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:12 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:07:12.709883+0000 mon.smithi053 (mon.0) 301 : audit [DBG] from='client.? 172.21.15.146:0/3588153620' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:07:14.140 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:07:14.140 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:07:15.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:15.369+0000 7fafacaa5700 1 -- 172.21.15.146:0/2485284408 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fafa80f51a0 msgr2=0x7fafa80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:15.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:15.369+0000 7fafacaa5700 1 --2- 172.21.15.146:0/2485284408 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fafa80f51a0 0x7fafa80f55c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7faf98009fb0 tx=0x7faf98009b30).stop 2024-01-21T20:07:15.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:15.369+0000 7fafacaa5700 1 -- 172.21.15.146:0/2485284408 shutdown_connections 2024-01-21T20:07:15.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:15.369+0000 7fafacaa5700 1 --2- 172.21.15.146:0/2485284408 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fafa80f51a0 0x7fafa80f55c0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:15.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:15.369+0000 7fafacaa5700 1 -- 172.21.15.146:0/2485284408 >> 172.21.15.146:0/2485284408 conn(0x7fafa80f09a0 msgr2=0x7fafa80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:15.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:15.369+0000 7fafacaa5700 1 -- 172.21.15.146:0/2485284408 shutdown_connections 2024-01-21T20:07:15.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:15.369+0000 7fafacaa5700 1 -- 172.21.15.146:0/2485284408 wait complete. 2024-01-21T20:07:15.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:15.369+0000 7fafacaa5700 1 Processor -- start 2024-01-21T20:07:15.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:15.369+0000 7fafacaa5700 1 -- start start 2024-01-21T20:07:15.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:15.370+0000 7fafacaa5700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fafa80f51a0 0x7fafa80fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:15.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:15.370+0000 7fafacaa5700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa80fb5a0 con 0x7fafa80f51a0 2024-01-21T20:07:15.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:15.370+0000 7fafa77fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fafa80f51a0 0x7fafa80fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:15.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:15.370+0000 7fafa77fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fafa80f51a0 0x7fafa80fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:59954/0 (socket says 172.21.15.146:59954) 2024-01-21T20:07:15.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:15.370+0000 7fafa77fe700 1 -- 172.21.15.146:0/1804941935 learned_addr learned my addr 172.21.15.146:0/1804941935 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:07:15.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:15.370+0000 7fafa77fe700 1 -- 172.21.15.146:0/1804941935 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf98005040 con 0x7fafa80f51a0 2024-01-21T20:07:15.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:15.370+0000 7fafa77fe700 1 --2- 172.21.15.146:0/1804941935 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fafa80f51a0 0x7fafa80fd120 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7faf98013510 tx=0x7faf9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:15.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:15.370+0000 7faf8ffff700 1 -- 172.21.15.146:0/1804941935 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf98009320 con 0x7fafa80f51a0 2024-01-21T20:07:15.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:15.371+0000 7fafacaa5700 1 -- 172.21.15.146:0/1804941935 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafa80fb740 con 0x7fafa80f51a0 2024-01-21T20:07:15.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:15.371+0000 7fafacaa5700 1 -- 172.21.15.146:0/1804941935 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafa80fbbd0 con 0x7fafa80f51a0 2024-01-21T20:07:15.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:15.371+0000 7faf8ffff700 1 -- 172.21.15.146:0/1804941935 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faf98013f00 con 0x7fafa80f51a0 2024-01-21T20:07:15.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:15.371+0000 7faf8ffff700 1 -- 172.21.15.146:0/1804941935 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf98014d90 con 0x7fafa80f51a0 2024-01-21T20:07:15.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:15.371+0000 7faf8ffff700 1 -- 172.21.15.146:0/1804941935 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faf98014420 con 0x7fafa80f51a0 2024-01-21T20:07:15.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:15.371+0000 7fafacaa5700 1 -- 172.21.15.146:0/1804941935 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf88004fa0 con 0x7fafa80f51a0 2024-01-21T20:07:15.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:15.372+0000 7faf8ffff700 1 --2- 172.21.15.146:0/1804941935 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7faf9002ffb0 0x7faf90032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:15.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:15.372+0000 7faf8ffff700 1 -- 172.21.15.146:0/1804941935 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faf98041280 con 0x7fafa80f51a0 2024-01-21T20:07:15.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:15.372+0000 7fafa6ffd700 1 --2- 172.21.15.146:0/1804941935 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7faf9002ffb0 0x7faf90032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:15.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:15.372+0000 7fafa6ffd700 1 --2- 172.21.15.146:0/1804941935 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7faf9002ffb0 0x7faf90032470 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7faf9c0096a0 tx=0x7faf9c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:15.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:15.376+0000 7faf8ffff700 1 -- 172.21.15.146:0/1804941935 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faf98012b20 con 0x7fafa80f51a0 2024-01-21T20:07:15.432 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:14 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:07:13.203116+0000 mgr.smithi053.fgsmol (mgr.14164) 250 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:15.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:15.579+0000 7fafacaa5700 1 -- 172.21.15.146:0/1804941935 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faf88005670 con 0x7fafa80f51a0 2024-01-21T20:07:15.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:15.580+0000 7faf8ffff700 1 -- 172.21.15.146:0/1804941935 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7faf98012b20 con 0x7fafa80f51a0 2024-01-21T20:07:15.570 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:07:15.570 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:07:15.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:15.581+0000 7fafacaa5700 1 -- 172.21.15.146:0/1804941935 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7faf9002ffb0 msgr2=0x7faf90032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:15.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:15.581+0000 7fafacaa5700 1 --2- 172.21.15.146:0/1804941935 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7faf9002ffb0 0x7faf90032470 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7faf9c0096a0 tx=0x7faf9c006b40).stop 2024-01-21T20:07:15.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:15.581+0000 7fafacaa5700 1 -- 172.21.15.146:0/1804941935 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fafa80f51a0 msgr2=0x7fafa80fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:15.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:15.581+0000 7fafacaa5700 1 --2- 172.21.15.146:0/1804941935 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fafa80f51a0 0x7fafa80fd120 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7faf98013510 tx=0x7faf9800b040).stop 2024-01-21T20:07:15.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:15.581+0000 7fafacaa5700 1 -- 172.21.15.146:0/1804941935 shutdown_connections 2024-01-21T20:07:15.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:15.581+0000 7fafacaa5700 1 --2- 172.21.15.146:0/1804941935 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7faf9002ffb0 0x7faf90032470 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:15.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:15.581+0000 7fafacaa5700 1 --2- 172.21.15.146:0/1804941935 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fafa80f51a0 0x7fafa80fd120 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:15.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:15.581+0000 7fafacaa5700 1 -- 172.21.15.146:0/1804941935 >> 172.21.15.146:0/1804941935 conn(0x7fafa80f09a0 msgr2=0x7fafa805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:15.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:15.581+0000 7fafacaa5700 1 -- 172.21.15.146:0/1804941935 shutdown_connections 2024-01-21T20:07:15.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:15.581+0000 7fafacaa5700 1 -- 172.21.15.146:0/1804941935 wait complete. 2024-01-21T20:07:15.573 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:07:16.434 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:15 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:07:15.581018+0000 mon.smithi053 (mon.0) 302 : audit [DBG] from='client.? 172.21.15.146:0/1804941935' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:07:17.048 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:07:17.049 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:07:17.433 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:07:15.203385+0000 mgr.smithi053.fgsmol (mgr.14164) 251 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:18.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:18.255+0000 7fca4f72f700 1 -- 172.21.15.146:0/143363814 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca480f5410 msgr2=0x7fca480f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:18.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:18.255+0000 7fca4f72f700 1 --2- 172.21.15.146:0/143363814 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca480f5410 0x7fca480f5830 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7fca44009fb0 tx=0x7fca44009b30).stop 2024-01-21T20:07:18.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:18.255+0000 7fca4f72f700 1 -- 172.21.15.146:0/143363814 shutdown_connections 2024-01-21T20:07:18.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:18.255+0000 7fca4f72f700 1 --2- 172.21.15.146:0/143363814 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca480f5410 0x7fca480f5830 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:18.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:18.255+0000 7fca4f72f700 1 -- 172.21.15.146:0/143363814 >> 172.21.15.146:0/143363814 conn(0x7fca480f0b70 msgr2=0x7fca480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:18.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:18.255+0000 7fca4f72f700 1 -- 172.21.15.146:0/143363814 shutdown_connections 2024-01-21T20:07:18.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:18.255+0000 7fca4f72f700 1 -- 172.21.15.146:0/143363814 wait complete. 2024-01-21T20:07:18.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:18.256+0000 7fca4f72f700 1 Processor -- start 2024-01-21T20:07:18.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:18.256+0000 7fca4f72f700 1 -- start start 2024-01-21T20:07:18.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:18.256+0000 7fca4f72f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca480f5410 0x7fca481093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:18.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:18.256+0000 7fca4f72f700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca48060280 con 0x7fca480f5410 2024-01-21T20:07:18.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:18.256+0000 7fca4d4cb700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca480f5410 0x7fca481093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:18.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:18.256+0000 7fca4d4cb700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca480f5410 0x7fca481093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:39712/0 (socket says 172.21.15.146:39712) 2024-01-21T20:07:18.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:18.256+0000 7fca4d4cb700 1 -- 172.21.15.146:0/2818627254 learned_addr learned my addr 172.21.15.146:0/2818627254 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:07:18.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:18.256+0000 7fca4d4cb700 1 -- 172.21.15.146:0/2818627254 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca44005040 con 0x7fca480f5410 2024-01-21T20:07:18.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:18.257+0000 7fca4d4cb700 1 --2- 172.21.15.146:0/2818627254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca480f5410 0x7fca481093f0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7fca44013510 tx=0x7fca4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:18.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:18.257+0000 7fca3dffb700 1 -- 172.21.15.146:0/2818627254 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca44009320 con 0x7fca480f5410 2024-01-21T20:07:18.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:18.257+0000 7fca4f72f700 1 -- 172.21.15.146:0/2818627254 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca48060480 con 0x7fca480f5410 2024-01-21T20:07:18.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:18.257+0000 7fca3dffb700 1 -- 172.21.15.146:0/2818627254 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fca44013f00 con 0x7fca480f5410 2024-01-21T20:07:18.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:18.257+0000 7fca3dffb700 1 -- 172.21.15.146:0/2818627254 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca44014d90 con 0x7fca480f5410 2024-01-21T20:07:18.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:18.257+0000 7fca4f72f700 1 -- 172.21.15.146:0/2818627254 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca48109b20 con 0x7fca480f5410 2024-01-21T20:07:18.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:18.258+0000 7fca3dffb700 1 -- 172.21.15.146:0/2818627254 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fca44014420 con 0x7fca480f5410 2024-01-21T20:07:18.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:18.258+0000 7fca4f72f700 1 -- 172.21.15.146:0/2818627254 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca48045ad0 con 0x7fca480f5410 2024-01-21T20:07:18.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:18.258+0000 7fca3dffb700 1 --2- 172.21.15.146:0/2818627254 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fca3402ff60 0x7fca34032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:18.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:18.258+0000 7fca3dffb700 1 -- 172.21.15.146:0/2818627254 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fca44041050 con 0x7fca480f5410 2024-01-21T20:07:18.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:18.258+0000 7fca4ccca700 1 --2- 172.21.15.146:0/2818627254 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fca3402ff60 0x7fca34032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:18.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:18.259+0000 7fca4ccca700 1 --2- 172.21.15.146:0/2818627254 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fca3402ff60 0x7fca34032420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fca380096a0 tx=0x7fca38006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:18.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:18.263+0000 7fca3dffb700 1 -- 172.21.15.146:0/2818627254 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fca440095f0 con 0x7fca480f5410 2024-01-21T20:07:18.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:18.467+0000 7fca4f72f700 1 -- 172.21.15.146:0/2818627254 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fca4810a340 con 0x7fca480f5410 2024-01-21T20:07:18.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:18.468+0000 7fca3dffb700 1 -- 172.21.15.146:0/2818627254 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fca4401db70 con 0x7fca480f5410 2024-01-21T20:07:18.458 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:07:18.459 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:07:18.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:18.469+0000 7fca4f72f700 1 -- 172.21.15.146:0/2818627254 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fca3402ff60 msgr2=0x7fca34032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:18.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:18.469+0000 7fca4f72f700 1 --2- 172.21.15.146:0/2818627254 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fca3402ff60 0x7fca34032420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fca380096a0 tx=0x7fca38006b40).stop 2024-01-21T20:07:18.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:18.469+0000 7fca4f72f700 1 -- 172.21.15.146:0/2818627254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca480f5410 msgr2=0x7fca481093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:18.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:18.469+0000 7fca4f72f700 1 --2- 172.21.15.146:0/2818627254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca480f5410 0x7fca481093f0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7fca44013510 tx=0x7fca4400b040).stop 2024-01-21T20:07:18.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:18.469+0000 7fca4f72f700 1 -- 172.21.15.146:0/2818627254 shutdown_connections 2024-01-21T20:07:18.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:18.469+0000 7fca4f72f700 1 --2- 172.21.15.146:0/2818627254 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fca3402ff60 0x7fca34032420 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:18.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:18.469+0000 7fca4f72f700 1 --2- 172.21.15.146:0/2818627254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca480f5410 0x7fca481093f0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:18.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:18.469+0000 7fca4f72f700 1 -- 172.21.15.146:0/2818627254 >> 172.21.15.146:0/2818627254 conn(0x7fca480f0b70 msgr2=0x7fca480f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:18.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:18.470+0000 7fca4f72f700 1 -- 172.21.15.146:0/2818627254 shutdown_connections 2024-01-21T20:07:18.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:18.470+0000 7fca4f72f700 1 -- 172.21.15.146:0/2818627254 wait complete. 2024-01-21T20:07:18.462 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:07:19.432 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:18 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:07:17.203746+0000 mgr.smithi053.fgsmol (mgr.14164) 252 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:19.433 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:18 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:07:18.469145+0000 mon.smithi053 (mon.0) 303 : audit [DBG] from='client.? 172.21.15.146:0/2818627254' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:07:19.844 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:07:19.844 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:07:21.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:21.038+0000 7fb782897700 1 -- 172.21.15.146:0/2947304199 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb77c0f5410 msgr2=0x7fb77c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:21.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:21.038+0000 7fb782897700 1 --2- 172.21.15.146:0/2947304199 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb77c0f5410 0x7fb77c0f5830 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7fb76c009fb0 tx=0x7fb76c009b30).stop 2024-01-21T20:07:21.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:21.038+0000 7fb782897700 1 -- 172.21.15.146:0/2947304199 shutdown_connections 2024-01-21T20:07:21.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:21.038+0000 7fb782897700 1 --2- 172.21.15.146:0/2947304199 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb77c0f5410 0x7fb77c0f5830 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:21.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:21.038+0000 7fb782897700 1 -- 172.21.15.146:0/2947304199 >> 172.21.15.146:0/2947304199 conn(0x7fb77c0f0b70 msgr2=0x7fb77c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:21.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:21.038+0000 7fb782897700 1 -- 172.21.15.146:0/2947304199 shutdown_connections 2024-01-21T20:07:21.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:21.038+0000 7fb782897700 1 -- 172.21.15.146:0/2947304199 wait complete. 2024-01-21T20:07:21.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:21.038+0000 7fb782897700 1 Processor -- start 2024-01-21T20:07:21.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:21.038+0000 7fb782897700 1 -- start start 2024-01-21T20:07:21.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:21.038+0000 7fb782897700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb77c0f5410 0x7fb77c109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:21.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:21.038+0000 7fb782897700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb77c060240 con 0x7fb77c0f5410 2024-01-21T20:07:21.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:21.039+0000 7fb77bfff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb77c0f5410 0x7fb77c109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:21.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:21.039+0000 7fb77bfff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb77c0f5410 0x7fb77c109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:39740/0 (socket says 172.21.15.146:39740) 2024-01-21T20:07:21.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:21.039+0000 7fb77bfff700 1 -- 172.21.15.146:0/2965586204 learned_addr learned my addr 172.21.15.146:0/2965586204 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:07:21.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:21.039+0000 7fb77bfff700 1 -- 172.21.15.146:0/2965586204 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb76c005040 con 0x7fb77c0f5410 2024-01-21T20:07:21.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:21.039+0000 7fb77bfff700 1 --2- 172.21.15.146:0/2965586204 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb77c0f5410 0x7fb77c109430 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7fb76c013510 tx=0x7fb76c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:21.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:21.039+0000 7fb7797fa700 1 -- 172.21.15.146:0/2965586204 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb76c009320 con 0x7fb77c0f5410 2024-01-21T20:07:21.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:21.039+0000 7fb782897700 1 -- 172.21.15.146:0/2965586204 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb77c060440 con 0x7fb77c0f5410 2024-01-21T20:07:21.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:21.039+0000 7fb7797fa700 1 -- 172.21.15.146:0/2965586204 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb76c013f00 con 0x7fb77c0f5410 2024-01-21T20:07:21.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:21.039+0000 7fb7797fa700 1 -- 172.21.15.146:0/2965586204 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb76c014d90 con 0x7fb77c0f5410 2024-01-21T20:07:21.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:21.039+0000 7fb782897700 1 -- 172.21.15.146:0/2965586204 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb77c109ae0 con 0x7fb77c0f5410 2024-01-21T20:07:21.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:21.040+0000 7fb7797fa700 1 -- 172.21.15.146:0/2965586204 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb76c0095f0 con 0x7fb77c0f5410 2024-01-21T20:07:21.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:21.040+0000 7fb782897700 1 -- 172.21.15.146:0/2965586204 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb77c045ad0 con 0x7fb77c0f5410 2024-01-21T20:07:21.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:21.040+0000 7fb7797fa700 1 --2- 172.21.15.146:0/2965586204 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb76802ff60 0x7fb768032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:21.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:21.040+0000 7fb7797fa700 1 -- 172.21.15.146:0/2965586204 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb76c040fe0 con 0x7fb77c0f5410 2024-01-21T20:07:21.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:21.041+0000 7fb773fff700 1 --2- 172.21.15.146:0/2965586204 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb76802ff60 0x7fb768032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:21.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:21.041+0000 7fb773fff700 1 --2- 172.21.15.146:0/2965586204 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb76802ff60 0x7fb768032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fb7640096a0 tx=0x7fb764006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:21.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:21.045+0000 7fb7797fa700 1 -- 172.21.15.146:0/2965586204 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb76c01d7f0 con 0x7fb77c0f5410 2024-01-21T20:07:21.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:21.251+0000 7fb782897700 1 -- 172.21.15.146:0/2965586204 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb77c0fae60 con 0x7fb77c0f5410 2024-01-21T20:07:21.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:21.252+0000 7fb7797fa700 1 -- 172.21.15.146:0/2965586204 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb76c010230 con 0x7fb77c0f5410 2024-01-21T20:07:21.243 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:07:21.243 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:07:21.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:21.253+0000 7fb782897700 1 -- 172.21.15.146:0/2965586204 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb76802ff60 msgr2=0x7fb768032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:21.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:21.253+0000 7fb782897700 1 --2- 172.21.15.146:0/2965586204 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb76802ff60 0x7fb768032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fb7640096a0 tx=0x7fb764006b40).stop 2024-01-21T20:07:21.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:21.253+0000 7fb782897700 1 -- 172.21.15.146:0/2965586204 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb77c0f5410 msgr2=0x7fb77c109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:21.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:21.253+0000 7fb782897700 1 --2- 172.21.15.146:0/2965586204 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb77c0f5410 0x7fb77c109430 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7fb76c013510 tx=0x7fb76c00b040).stop 2024-01-21T20:07:21.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:21.253+0000 7fb782897700 1 -- 172.21.15.146:0/2965586204 shutdown_connections 2024-01-21T20:07:21.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:21.253+0000 7fb782897700 1 --2- 172.21.15.146:0/2965586204 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fb76802ff60 0x7fb768032420 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:21.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:21.253+0000 7fb782897700 1 --2- 172.21.15.146:0/2965586204 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb77c0f5410 0x7fb77c109430 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:21.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:21.254+0000 7fb782897700 1 -- 172.21.15.146:0/2965586204 >> 172.21.15.146:0/2965586204 conn(0x7fb77c0f0b70 msgr2=0x7fb77c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:21.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:21.254+0000 7fb782897700 1 -- 172.21.15.146:0/2965586204 shutdown_connections 2024-01-21T20:07:21.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:21.254+0000 7fb782897700 1 -- 172.21.15.146:0/2965586204 wait complete. 2024-01-21T20:07:21.245 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:07:21.433 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:20 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:07:19.204114+0000 mgr.smithi053.fgsmol (mgr.14164) 253 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:22.432 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:21 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:07:21.253514+0000 mon.smithi053 (mon.0) 304 : audit [DBG] from='client.? 172.21.15.146:0/2965586204' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:07:22.670 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:07:22.670 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:07:23.433 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:07:21.204406+0000 mgr.smithi053.fgsmol (mgr.14164) 254 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:23.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:23.916+0000 7ff71401e700 1 -- 172.21.15.146:0/2066343909 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff70c0f5410 msgr2=0x7ff70c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:23.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:23.916+0000 7ff71401e700 1 --2- 172.21.15.146:0/2066343909 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff70c0f5410 0x7ff70c0f5830 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7ff6fc009fb0 tx=0x7ff6fc009b30).stop 2024-01-21T20:07:23.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:23.916+0000 7ff71401e700 1 -- 172.21.15.146:0/2066343909 shutdown_connections 2024-01-21T20:07:23.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:23.916+0000 7ff71401e700 1 --2- 172.21.15.146:0/2066343909 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff70c0f5410 0x7ff70c0f5830 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:23.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:23.916+0000 7ff71401e700 1 -- 172.21.15.146:0/2066343909 >> 172.21.15.146:0/2066343909 conn(0x7ff70c0f0b70 msgr2=0x7ff70c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:23.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:23.916+0000 7ff71401e700 1 -- 172.21.15.146:0/2066343909 shutdown_connections 2024-01-21T20:07:23.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:23.916+0000 7ff71401e700 1 -- 172.21.15.146:0/2066343909 wait complete. 2024-01-21T20:07:23.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:23.917+0000 7ff71401e700 1 Processor -- start 2024-01-21T20:07:23.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:23.917+0000 7ff71401e700 1 -- start start 2024-01-21T20:07:23.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:23.917+0000 7ff71401e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff70c0f5410 0x7ff70c109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:23.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:23.917+0000 7ff71401e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff70c060220 con 0x7ff70c0f5410 2024-01-21T20:07:23.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:23.917+0000 7ff711dba700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff70c0f5410 0x7ff70c109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:23.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:23.917+0000 7ff711dba700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff70c0f5410 0x7ff70c109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:39750/0 (socket says 172.21.15.146:39750) 2024-01-21T20:07:23.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:23.917+0000 7ff711dba700 1 -- 172.21.15.146:0/1036486136 learned_addr learned my addr 172.21.15.146:0/1036486136 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:07:23.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:23.917+0000 7ff711dba700 1 -- 172.21.15.146:0/1036486136 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6fc005040 con 0x7ff70c0f5410 2024-01-21T20:07:23.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:23.917+0000 7ff711dba700 1 --2- 172.21.15.146:0/1036486136 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff70c0f5410 0x7ff70c109450 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7ff6fc013510 tx=0x7ff6fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:23.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:23.918+0000 7ff7027fc700 1 -- 172.21.15.146:0/1036486136 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6fc009320 con 0x7ff70c0f5410 2024-01-21T20:07:23.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:23.918+0000 7ff71401e700 1 -- 172.21.15.146:0/1036486136 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff70c060420 con 0x7ff70c0f5410 2024-01-21T20:07:23.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:23.918+0000 7ff7027fc700 1 -- 172.21.15.146:0/1036486136 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff6fc013f00 con 0x7ff70c0f5410 2024-01-21T20:07:23.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:23.918+0000 7ff7027fc700 1 -- 172.21.15.146:0/1036486136 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6fc014d90 con 0x7ff70c0f5410 2024-01-21T20:07:23.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:23.918+0000 7ff71401e700 1 -- 172.21.15.146:0/1036486136 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff70c109b00 con 0x7ff70c0f5410 2024-01-21T20:07:23.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:23.919+0000 7ff7027fc700 1 -- 172.21.15.146:0/1036486136 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff6fc014420 con 0x7ff70c0f5410 2024-01-21T20:07:23.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:23.919+0000 7ff7027fc700 1 --2- 172.21.15.146:0/1036486136 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff6f802ff60 0x7ff6f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:23.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:23.919+0000 7ff71401e700 1 -- 172.21.15.146:0/1036486136 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff70c059100 con 0x7ff70c0f5410 2024-01-21T20:07:23.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:23.919+0000 7ff7027fc700 1 -- 172.21.15.146:0/1036486136 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff6fc041080 con 0x7ff70c0f5410 2024-01-21T20:07:23.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:23.919+0000 7ff7115b9700 1 --2- 172.21.15.146:0/1036486136 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff6f802ff60 0x7ff6f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:23.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:23.919+0000 7ff7115b9700 1 --2- 172.21.15.146:0/1036486136 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff6f802ff60 0x7ff6f8032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7ff7080096a0 tx=0x7ff708006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:23.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:23.924+0000 7ff7027fc700 1 -- 172.21.15.146:0/1036486136 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff6fc0095f0 con 0x7ff70c0f5410 2024-01-21T20:07:24.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:24.130+0000 7ff71401e700 1 -- 172.21.15.146:0/1036486136 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff70c045ad0 con 0x7ff70c0f5410 2024-01-21T20:07:24.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:24.131+0000 7ff7027fc700 1 -- 172.21.15.146:0/1036486136 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff6fc01db70 con 0x7ff70c0f5410 2024-01-21T20:07:24.121 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:07:24.121 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:07:24.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:24.132+0000 7ff71401e700 1 -- 172.21.15.146:0/1036486136 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff6f802ff60 msgr2=0x7ff6f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:24.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:24.132+0000 7ff71401e700 1 --2- 172.21.15.146:0/1036486136 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff6f802ff60 0x7ff6f8032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7ff7080096a0 tx=0x7ff708006b40).stop 2024-01-21T20:07:24.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:24.132+0000 7ff71401e700 1 -- 172.21.15.146:0/1036486136 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff70c0f5410 msgr2=0x7ff70c109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:24.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:24.132+0000 7ff71401e700 1 --2- 172.21.15.146:0/1036486136 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff70c0f5410 0x7ff70c109450 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7ff6fc013510 tx=0x7ff6fc00b040).stop 2024-01-21T20:07:24.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:24.132+0000 7ff71401e700 1 -- 172.21.15.146:0/1036486136 shutdown_connections 2024-01-21T20:07:24.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:24.132+0000 7ff71401e700 1 --2- 172.21.15.146:0/1036486136 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff6f802ff60 0x7ff6f8032420 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:24.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:24.132+0000 7ff71401e700 1 --2- 172.21.15.146:0/1036486136 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff70c0f5410 0x7ff70c109450 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:24.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:24.132+0000 7ff71401e700 1 -- 172.21.15.146:0/1036486136 >> 172.21.15.146:0/1036486136 conn(0x7ff70c0f0b70 msgr2=0x7ff70c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:24.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:24.132+0000 7ff71401e700 1 -- 172.21.15.146:0/1036486136 shutdown_connections 2024-01-21T20:07:24.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:24.132+0000 7ff71401e700 1 -- 172.21.15.146:0/1036486136 wait complete. 2024-01-21T20:07:24.124 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:07:25.433 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:24 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:07:23.204661+0000 mgr.smithi053.fgsmol (mgr.14164) 255 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:25.433 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:24 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:07:24.132023+0000 mon.smithi053 (mon.0) 305 : audit [DBG] from='client.? 172.21.15.146:0/1036486136' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:07:25.554 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:07:25.554 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:07:26.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:26.725+0000 7f9a4d684700 1 -- 172.21.15.146:0/3846259689 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a480f51a0 msgr2=0x7f9a480f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:26.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:26.725+0000 7f9a4d684700 1 --2- 172.21.15.146:0/3846259689 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a480f51a0 0x7f9a480f55c0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f9a38009fb0 tx=0x7f9a38009b30).stop 2024-01-21T20:07:26.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:26.725+0000 7f9a4d684700 1 -- 172.21.15.146:0/3846259689 shutdown_connections 2024-01-21T20:07:26.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:26.725+0000 7f9a4d684700 1 --2- 172.21.15.146:0/3846259689 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a480f51a0 0x7f9a480f55c0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:26.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:26.725+0000 7f9a4d684700 1 -- 172.21.15.146:0/3846259689 >> 172.21.15.146:0/3846259689 conn(0x7f9a480f09a0 msgr2=0x7f9a480f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:26.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:26.725+0000 7f9a4d684700 1 -- 172.21.15.146:0/3846259689 shutdown_connections 2024-01-21T20:07:26.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:26.725+0000 7f9a4d684700 1 -- 172.21.15.146:0/3846259689 wait complete. 2024-01-21T20:07:26.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:26.725+0000 7f9a4d684700 1 Processor -- start 2024-01-21T20:07:26.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:26.726+0000 7f9a4d684700 1 -- start start 2024-01-21T20:07:26.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:26.726+0000 7f9a4d684700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a480f51a0 0x7f9a481053e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:26.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:26.726+0000 7f9a4d684700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a481058d0 con 0x7f9a480f51a0 2024-01-21T20:07:26.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:26.726+0000 7f9a47fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a480f51a0 0x7f9a481053e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:26.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:26.726+0000 7f9a47fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a480f51a0 0x7f9a481053e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:39776/0 (socket says 172.21.15.146:39776) 2024-01-21T20:07:26.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:26.726+0000 7f9a47fff700 1 -- 172.21.15.146:0/3141614270 learned_addr learned my addr 172.21.15.146:0/3141614270 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:07:26.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:26.726+0000 7f9a47fff700 1 -- 172.21.15.146:0/3141614270 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a38005040 con 0x7f9a480f51a0 2024-01-21T20:07:26.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:26.726+0000 7f9a47fff700 1 --2- 172.21.15.146:0/3141614270 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a480f51a0 0x7f9a481053e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f9a38013510 tx=0x7f9a3800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:26.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:26.726+0000 7f9a44ff9700 1 -- 172.21.15.146:0/3141614270 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a38009320 con 0x7f9a480f51a0 2024-01-21T20:07:26.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:26.726+0000 7f9a4d684700 1 -- 172.21.15.146:0/3141614270 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a48105ad0 con 0x7f9a480f51a0 2024-01-21T20:07:26.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:26.727+0000 7f9a44ff9700 1 -- 172.21.15.146:0/3141614270 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9a38013f00 con 0x7f9a480f51a0 2024-01-21T20:07:26.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:26.727+0000 7f9a4d684700 1 -- 172.21.15.146:0/3141614270 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a48105f60 con 0x7f9a480f51a0 2024-01-21T20:07:26.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:26.727+0000 7f9a44ff9700 1 -- 172.21.15.146:0/3141614270 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a38014dc0 con 0x7f9a480f51a0 2024-01-21T20:07:26.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:26.727+0000 7f9a4d684700 1 -- 172.21.15.146:0/3141614270 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a48045ad0 con 0x7f9a480f51a0 2024-01-21T20:07:26.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:26.727+0000 7f9a44ff9700 1 -- 172.21.15.146:0/3141614270 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9a380095f0 con 0x7f9a480f51a0 2024-01-21T20:07:26.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:26.728+0000 7f9a44ff9700 1 --2- 172.21.15.146:0/3141614270 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f9a3002ff60 0x7f9a30032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:26.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:26.728+0000 7f9a44ff9700 1 -- 172.21.15.146:0/3141614270 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9a38041080 con 0x7f9a480f51a0 2024-01-21T20:07:26.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:26.728+0000 7f9a477fe700 1 --2- 172.21.15.146:0/3141614270 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f9a3002ff60 0x7f9a30032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:26.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:26.728+0000 7f9a477fe700 1 --2- 172.21.15.146:0/3141614270 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f9a3002ff60 0x7f9a30032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f9a3c0096a0 tx=0x7f9a3c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:26.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:26.732+0000 7f9a44ff9700 1 -- 172.21.15.146:0/3141614270 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9a38014420 con 0x7f9a480f51a0 2024-01-21T20:07:26.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:26.940+0000 7f9a4d684700 1 -- 172.21.15.146:0/3141614270 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9a480fabe0 con 0x7f9a480f51a0 2024-01-21T20:07:26.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:26.941+0000 7f9a44ff9700 1 -- 172.21.15.146:0/3141614270 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9a3801c790 con 0x7f9a480f51a0 2024-01-21T20:07:26.931 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:07:26.931 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:07:26.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:26.942+0000 7f9a4d684700 1 -- 172.21.15.146:0/3141614270 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f9a3002ff60 msgr2=0x7f9a30032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:26.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:26.943+0000 7f9a4d684700 1 --2- 172.21.15.146:0/3141614270 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f9a3002ff60 0x7f9a30032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f9a3c0096a0 tx=0x7f9a3c006b40).stop 2024-01-21T20:07:26.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:26.943+0000 7f9a4d684700 1 -- 172.21.15.146:0/3141614270 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a480f51a0 msgr2=0x7f9a481053e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:26.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:26.943+0000 7f9a4d684700 1 --2- 172.21.15.146:0/3141614270 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a480f51a0 0x7f9a481053e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f9a38013510 tx=0x7f9a3800b040).stop 2024-01-21T20:07:26.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:26.943+0000 7f9a4d684700 1 -- 172.21.15.146:0/3141614270 shutdown_connections 2024-01-21T20:07:26.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:26.943+0000 7f9a4d684700 1 --2- 172.21.15.146:0/3141614270 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f9a3002ff60 0x7f9a30032420 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:26.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:26.943+0000 7f9a4d684700 1 --2- 172.21.15.146:0/3141614270 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a480f51a0 0x7f9a481053e0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:26.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:26.943+0000 7f9a4d684700 1 -- 172.21.15.146:0/3141614270 >> 172.21.15.146:0/3141614270 conn(0x7f9a480f09a0 msgr2=0x7f9a480f13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:26.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:26.943+0000 7f9a4d684700 1 -- 172.21.15.146:0/3141614270 shutdown_connections 2024-01-21T20:07:26.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:26.943+0000 7f9a4d684700 1 -- 172.21.15.146:0/3141614270 wait complete. 2024-01-21T20:07:26.934 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:07:27.264 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:26 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:07:25.204907+0000 mgr.smithi053.fgsmol (mgr.14164) 256 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:27.264 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:26 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:07:26.941905+0000 mon.smithi053 (mon.0) 306 : audit [DBG] from='client.? 172.21.15.146:0/3141614270' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:07:28.376 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:07:28.376 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:07:29.433 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:28 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:07:27.205155+0000 mgr.smithi053.fgsmol (mgr.14164) 257 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:29.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:29.609+0000 7f2ddca97700 1 -- 172.21.15.146:0/2219931479 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2dd80f6510 msgr2=0x7f2dd80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:29.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:29.609+0000 7f2ddca97700 1 --2- 172.21.15.146:0/2219931479 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2dd80f6510 0x7f2dd80f6930 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f2dc8009fb0 tx=0x7f2dc8009b30).stop 2024-01-21T20:07:29.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:29.609+0000 7f2ddca97700 1 -- 172.21.15.146:0/2219931479 shutdown_connections 2024-01-21T20:07:29.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:29.609+0000 7f2ddca97700 1 --2- 172.21.15.146:0/2219931479 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2dd80f6510 0x7f2dd80f6930 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:29.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:29.609+0000 7f2ddca97700 1 -- 172.21.15.146:0/2219931479 >> 172.21.15.146:0/2219931479 conn(0x7f2dd80f1c70 msgr2=0x7f2dd80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:29.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:29.609+0000 7f2ddca97700 1 -- 172.21.15.146:0/2219931479 shutdown_connections 2024-01-21T20:07:29.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:29.609+0000 7f2ddca97700 1 -- 172.21.15.146:0/2219931479 wait complete. 2024-01-21T20:07:29.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:29.610+0000 7f2ddca97700 1 Processor -- start 2024-01-21T20:07:29.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:29.610+0000 7f2ddca97700 1 -- start start 2024-01-21T20:07:29.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:29.610+0000 7f2ddca97700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2dd80f6510 0x7f2dd8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:29.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:29.610+0000 7f2ddca97700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd8106c20 con 0x7f2dd80f6510 2024-01-21T20:07:29.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:29.610+0000 7f2dd659c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2dd80f6510 0x7f2dd8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:29.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:29.610+0000 7f2dd659c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2dd80f6510 0x7f2dd8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:49080/0 (socket says 172.21.15.146:49080) 2024-01-21T20:07:29.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:29.610+0000 7f2dd659c700 1 -- 172.21.15.146:0/373395171 learned_addr learned my addr 172.21.15.146:0/373395171 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:07:29.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:29.610+0000 7f2dd659c700 1 -- 172.21.15.146:0/373395171 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc8005040 con 0x7f2dd80f6510 2024-01-21T20:07:29.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:29.611+0000 7f2dd659c700 1 --2- 172.21.15.146:0/373395171 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2dd80f6510 0x7f2dd8106730 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f2dc8014000 tx=0x7f2dc800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:29.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:29.611+0000 7f2dceffd700 1 -- 172.21.15.146:0/373395171 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2dc80095f0 con 0x7f2dd80f6510 2024-01-21T20:07:29.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:29.611+0000 7f2dceffd700 1 -- 172.21.15.146:0/373395171 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2dc8007e70 con 0x7f2dd80f6510 2024-01-21T20:07:29.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:29.611+0000 7f2ddca97700 1 -- 172.21.15.146:0/373395171 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dd8106e20 con 0x7f2dd80f6510 2024-01-21T20:07:29.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:29.611+0000 7f2dceffd700 1 -- 172.21.15.146:0/373395171 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2dc8014ea0 con 0x7f2dd80f6510 2024-01-21T20:07:29.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:29.611+0000 7f2ddca97700 1 -- 172.21.15.146:0/373395171 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dd81072b0 con 0x7f2dd80f6510 2024-01-21T20:07:29.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:29.612+0000 7f2dceffd700 1 -- 172.21.15.146:0/373395171 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2dc8025070 con 0x7f2dd80f6510 2024-01-21T20:07:29.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:29.612+0000 7f2ddca97700 1 -- 172.21.15.146:0/373395171 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dd8045ad0 con 0x7f2dd80f6510 2024-01-21T20:07:29.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:29.612+0000 7f2dceffd700 1 --2- 172.21.15.146:0/373395171 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f2dc402ff60 0x7f2dc4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:29.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:29.612+0000 7f2dceffd700 1 -- 172.21.15.146:0/373395171 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2dc8040a30 con 0x7f2dd80f6510 2024-01-21T20:07:29.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:29.612+0000 7f2dd5d9b700 1 --2- 172.21.15.146:0/373395171 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f2dc402ff60 0x7f2dc4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:29.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:29.612+0000 7f2dd5d9b700 1 --2- 172.21.15.146:0/373395171 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f2dc402ff60 0x7f2dc4032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f2dc00096a0 tx=0x7f2dc0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:29.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:29.617+0000 7f2dceffd700 1 -- 172.21.15.146:0/373395171 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2dc80079c0 con 0x7f2dd80f6510 2024-01-21T20:07:29.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:29.819+0000 7f2ddca97700 1 -- 172.21.15.146:0/373395171 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2dd80fbf60 con 0x7f2dd80f6510 2024-01-21T20:07:29.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:29.819+0000 7f2dceffd700 1 -- 172.21.15.146:0/373395171 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2dc800f300 con 0x7f2dd80f6510 2024-01-21T20:07:29.810 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:07:29.810 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:07:29.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:29.821+0000 7f2ddca97700 1 -- 172.21.15.146:0/373395171 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f2dc402ff60 msgr2=0x7f2dc4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:29.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:29.821+0000 7f2ddca97700 1 --2- 172.21.15.146:0/373395171 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f2dc402ff60 0x7f2dc4032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f2dc00096a0 tx=0x7f2dc0006b40).stop 2024-01-21T20:07:29.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:29.821+0000 7f2ddca97700 1 -- 172.21.15.146:0/373395171 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2dd80f6510 msgr2=0x7f2dd8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:29.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:29.821+0000 7f2ddca97700 1 --2- 172.21.15.146:0/373395171 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2dd80f6510 0x7f2dd8106730 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f2dc8014000 tx=0x7f2dc800b040).stop 2024-01-21T20:07:29.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:29.821+0000 7f2ddca97700 1 -- 172.21.15.146:0/373395171 shutdown_connections 2024-01-21T20:07:29.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:29.821+0000 7f2ddca97700 1 --2- 172.21.15.146:0/373395171 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f2dc402ff60 0x7f2dc4032420 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:29.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:29.821+0000 7f2ddca97700 1 --2- 172.21.15.146:0/373395171 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2dd80f6510 0x7f2dd8106730 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:29.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:29.821+0000 7f2ddca97700 1 -- 172.21.15.146:0/373395171 >> 172.21.15.146:0/373395171 conn(0x7f2dd80f1c70 msgr2=0x7f2dd80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:29.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:29.821+0000 7f2ddca97700 1 -- 172.21.15.146:0/373395171 shutdown_connections 2024-01-21T20:07:29.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:29.821+0000 7f2ddca97700 1 -- 172.21.15.146:0/373395171 wait complete. 2024-01-21T20:07:29.813 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:07:30.433 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:29 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:07:29.820832+0000 mon.smithi053 (mon.0) 307 : audit [DBG] from='client.? 172.21.15.146:0/373395171' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:07:31.209 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:07:31.210 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:07:31.432 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:30 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:07:29.205512+0000 mgr.smithi053.fgsmol (mgr.14164) 258 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:32.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:32.434+0000 7f2e188ad700 1 -- 172.21.15.146:0/3831559805 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e140f51a0 msgr2=0x7f2e140f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:32.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:32.434+0000 7f2e188ad700 1 --2- 172.21.15.146:0/3831559805 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e140f51a0 0x7f2e140f55c0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f2e04009960 tx=0x7f2e04004d10).stop 2024-01-21T20:07:32.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:32.434+0000 7f2e188ad700 1 -- 172.21.15.146:0/3831559805 shutdown_connections 2024-01-21T20:07:32.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:32.434+0000 7f2e188ad700 1 --2- 172.21.15.146:0/3831559805 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e140f51a0 0x7f2e140f55c0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:32.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:32.434+0000 7f2e188ad700 1 -- 172.21.15.146:0/3831559805 >> 172.21.15.146:0/3831559805 conn(0x7f2e140f09a0 msgr2=0x7f2e140f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:32.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:32.434+0000 7f2e188ad700 1 -- 172.21.15.146:0/3831559805 shutdown_connections 2024-01-21T20:07:32.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:32.435+0000 7f2e188ad700 1 -- 172.21.15.146:0/3831559805 wait complete. 2024-01-21T20:07:32.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:32.435+0000 7f2e188ad700 1 Processor -- start 2024-01-21T20:07:32.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:32.435+0000 7f2e188ad700 1 -- start start 2024-01-21T20:07:32.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:32.435+0000 7f2e188ad700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e140f51a0 0x7f2e1410b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:32.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:32.435+0000 7f2e188ad700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e14106fc0 con 0x7f2e140f51a0 2024-01-21T20:07:32.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:32.435+0000 7f2e12d9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e140f51a0 0x7f2e1410b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:32.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:32.436+0000 7f2e12d9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e140f51a0 0x7f2e1410b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:49102/0 (socket says 172.21.15.146:49102) 2024-01-21T20:07:32.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:32.436+0000 7f2e12d9d700 1 -- 172.21.15.146:0/3505732389 learned_addr learned my addr 172.21.15.146:0/3505732389 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:07:32.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:32.436+0000 7f2e12d9d700 1 -- 172.21.15.146:0/3505732389 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e04009640 con 0x7f2e140f51a0 2024-01-21T20:07:32.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:32.436+0000 7f2e12d9d700 1 --2- 172.21.15.146:0/3505732389 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e140f51a0 0x7f2e1410b5d0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f2e04007bd0 tx=0x7f2e04004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:32.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:32.436+0000 7f2e0b7fe700 1 -- 172.21.15.146:0/3505732389 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e04018070 con 0x7f2e140f51a0 2024-01-21T20:07:32.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:32.436+0000 7f2e188ad700 1 -- 172.21.15.146:0/3505732389 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e14060220 con 0x7f2e140f51a0 2024-01-21T20:07:32.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:32.436+0000 7f2e0b7fe700 1 -- 172.21.15.146:0/3505732389 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2e0400b870 con 0x7f2e140f51a0 2024-01-21T20:07:32.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:32.436+0000 7f2e0b7fe700 1 -- 172.21.15.146:0/3505732389 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e0401d420 con 0x7f2e140f51a0 2024-01-21T20:07:32.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:32.436+0000 7f2e188ad700 1 -- 172.21.15.146:0/3505732389 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e1410bac0 con 0x7f2e140f51a0 2024-01-21T20:07:32.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:32.437+0000 7f2e0b7fe700 1 -- 172.21.15.146:0/3505732389 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2e04024060 con 0x7f2e140f51a0 2024-01-21T20:07:32.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:32.437+0000 7f2e188ad700 1 -- 172.21.15.146:0/3505732389 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e14045ad0 con 0x7f2e140f51a0 2024-01-21T20:07:32.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:32.437+0000 7f2e0b7fe700 1 --2- 172.21.15.146:0/3505732389 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f2e0002ff60 0x7f2e00032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:32.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:32.437+0000 7f2e0b7fe700 1 -- 172.21.15.146:0/3505732389 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2e04041150 con 0x7f2e140f51a0 2024-01-21T20:07:32.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:32.437+0000 7f2e1259c700 1 --2- 172.21.15.146:0/3505732389 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f2e0002ff60 0x7f2e00032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:32.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:32.438+0000 7f2e1259c700 1 --2- 172.21.15.146:0/3505732389 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f2e0002ff60 0x7f2e00032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f2dfc0096a0 tx=0x7f2dfc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:32.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:32.442+0000 7f2e0b7fe700 1 -- 172.21.15.146:0/3505732389 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2e04025520 con 0x7f2e140f51a0 2024-01-21T20:07:32.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:32.646+0000 7f2e188ad700 1 -- 172.21.15.146:0/3505732389 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2e1410c280 con 0x7f2e140f51a0 2024-01-21T20:07:32.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:32.647+0000 7f2e0b7fe700 1 -- 172.21.15.146:0/3505732389 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2e0401a2d0 con 0x7f2e140f51a0 2024-01-21T20:07:32.638 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:07:32.638 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:07:32.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:32.648+0000 7f2e188ad700 1 -- 172.21.15.146:0/3505732389 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f2e0002ff60 msgr2=0x7f2e00032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:32.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:32.648+0000 7f2e188ad700 1 --2- 172.21.15.146:0/3505732389 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f2e0002ff60 0x7f2e00032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f2dfc0096a0 tx=0x7f2dfc006b40).stop 2024-01-21T20:07:32.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:32.648+0000 7f2e188ad700 1 -- 172.21.15.146:0/3505732389 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e140f51a0 msgr2=0x7f2e1410b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:32.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:32.648+0000 7f2e188ad700 1 --2- 172.21.15.146:0/3505732389 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e140f51a0 0x7f2e1410b5d0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f2e04007bd0 tx=0x7f2e04004d10).stop 2024-01-21T20:07:32.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:32.649+0000 7f2e188ad700 1 -- 172.21.15.146:0/3505732389 shutdown_connections 2024-01-21T20:07:32.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:32.649+0000 7f2e188ad700 1 --2- 172.21.15.146:0/3505732389 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f2e0002ff60 0x7f2e00032420 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:32.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:32.649+0000 7f2e188ad700 1 --2- 172.21.15.146:0/3505732389 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e140f51a0 0x7f2e1410b5d0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:32.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:32.649+0000 7f2e188ad700 1 -- 172.21.15.146:0/3505732389 >> 172.21.15.146:0/3505732389 conn(0x7f2e140f09a0 msgr2=0x7f2e140f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:32.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:32.649+0000 7f2e188ad700 1 -- 172.21.15.146:0/3505732389 shutdown_connections 2024-01-21T20:07:32.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:32.649+0000 7f2e188ad700 1 -- 172.21.15.146:0/3505732389 wait complete. 2024-01-21T20:07:32.640 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:07:33.433 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:32 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:07:31.205862+0000 mgr.smithi053.fgsmol (mgr.14164) 259 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:33.433 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:32 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:07:32.648468+0000 mon.smithi053 (mon.0) 308 : audit [DBG] from='client.? 172.21.15.146:0/3505732389' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:07:34.073 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:07:34.073 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:07:35.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:35.286+0000 7ff22b16a700 1 -- 172.21.15.146:0/2555260988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2240f62a0 msgr2=0x7ff2240f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:35.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:35.286+0000 7ff22b16a700 1 --2- 172.21.15.146:0/2555260988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2240f62a0 0x7ff2240f66c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7ff220009fb0 tx=0x7ff220009b30).stop 2024-01-21T20:07:35.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:35.286+0000 7ff22b16a700 1 -- 172.21.15.146:0/2555260988 shutdown_connections 2024-01-21T20:07:35.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:35.286+0000 7ff22b16a700 1 --2- 172.21.15.146:0/2555260988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2240f62a0 0x7ff2240f66c0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:35.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:35.286+0000 7ff22b16a700 1 -- 172.21.15.146:0/2555260988 >> 172.21.15.146:0/2555260988 conn(0x7ff2240f1aa0 msgr2=0x7ff2240f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:35.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:35.286+0000 7ff22b16a700 1 -- 172.21.15.146:0/2555260988 shutdown_connections 2024-01-21T20:07:35.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:35.286+0000 7ff22b16a700 1 -- 172.21.15.146:0/2555260988 wait complete. 2024-01-21T20:07:35.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:35.287+0000 7ff22b16a700 1 Processor -- start 2024-01-21T20:07:35.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:35.287+0000 7ff22b16a700 1 -- start start 2024-01-21T20:07:35.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:35.287+0000 7ff22b16a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2240f62a0 0x7ff2241064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:35.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:35.287+0000 7ff22b16a700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2241069a0 con 0x7ff2240f62a0 2024-01-21T20:07:35.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:35.287+0000 7ff22a168700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2240f62a0 0x7ff2241064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:35.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:35.288+0000 7ff22a168700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2240f62a0 0x7ff2241064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:49124/0 (socket says 172.21.15.146:49124) 2024-01-21T20:07:35.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:35.288+0000 7ff22a168700 1 -- 172.21.15.146:0/3953306259 learned_addr learned my addr 172.21.15.146:0/3953306259 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:07:35.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:35.288+0000 7ff22a168700 1 -- 172.21.15.146:0/3953306259 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff220005040 con 0x7ff2240f62a0 2024-01-21T20:07:35.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:35.288+0000 7ff22a168700 1 --2- 172.21.15.146:0/3953306259 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2240f62a0 0x7ff2241064b0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7ff220013510 tx=0x7ff22000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:35.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:35.288+0000 7ff21affd700 1 -- 172.21.15.146:0/3953306259 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff220009320 con 0x7ff2240f62a0 2024-01-21T20:07:35.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:35.288+0000 7ff22b16a700 1 -- 172.21.15.146:0/3953306259 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff224106ba0 con 0x7ff2240f62a0 2024-01-21T20:07:35.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:35.288+0000 7ff21affd700 1 -- 172.21.15.146:0/3953306259 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff220013f00 con 0x7ff2240f62a0 2024-01-21T20:07:35.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:35.288+0000 7ff21affd700 1 -- 172.21.15.146:0/3953306259 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff220014dc0 con 0x7ff2240f62a0 2024-01-21T20:07:35.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:35.288+0000 7ff22b16a700 1 -- 172.21.15.146:0/3953306259 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff224107030 con 0x7ff2240f62a0 2024-01-21T20:07:35.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:35.289+0000 7ff21affd700 1 -- 172.21.15.146:0/3953306259 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff2200095f0 con 0x7ff2240f62a0 2024-01-21T20:07:35.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:35.289+0000 7ff21affd700 1 --2- 172.21.15.146:0/3953306259 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff210030000 0x7ff2100324c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:35.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:35.289+0000 7ff22b16a700 1 -- 172.21.15.146:0/3953306259 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff224045ad0 con 0x7ff2240f62a0 2024-01-21T20:07:35.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:35.289+0000 7ff21affd700 1 -- 172.21.15.146:0/3953306259 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff220041070 con 0x7ff2240f62a0 2024-01-21T20:07:35.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:35.289+0000 7ff229967700 1 --2- 172.21.15.146:0/3953306259 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff210030000 0x7ff2100324c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:35.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:35.290+0000 7ff229967700 1 --2- 172.21.15.146:0/3953306259 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff210030000 0x7ff2100324c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7ff2140096a0 tx=0x7ff214006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:35.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:35.294+0000 7ff21affd700 1 -- 172.21.15.146:0/3953306259 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff220010230 con 0x7ff2240f62a0 2024-01-21T20:07:35.432 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:34 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:07:33.206228+0000 mgr.smithi053.fgsmol (mgr.14164) 260 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:35.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:35.499+0000 7ff22b16a700 1 -- 172.21.15.146:0/3953306259 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff224107ee0 con 0x7ff2240f62a0 2024-01-21T20:07:35.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:35.499+0000 7ff21affd700 1 -- 172.21.15.146:0/3953306259 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff220010230 con 0x7ff2240f62a0 2024-01-21T20:07:35.490 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:07:35.490 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:07:35.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:35.501+0000 7ff22b16a700 1 -- 172.21.15.146:0/3953306259 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff210030000 msgr2=0x7ff2100324c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:35.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:35.501+0000 7ff22b16a700 1 --2- 172.21.15.146:0/3953306259 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff210030000 0x7ff2100324c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7ff2140096a0 tx=0x7ff214006b40).stop 2024-01-21T20:07:35.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:35.501+0000 7ff22b16a700 1 -- 172.21.15.146:0/3953306259 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2240f62a0 msgr2=0x7ff2241064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:35.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:35.501+0000 7ff22b16a700 1 --2- 172.21.15.146:0/3953306259 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2240f62a0 0x7ff2241064b0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7ff220013510 tx=0x7ff22000b040).stop 2024-01-21T20:07:35.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:35.501+0000 7ff22b16a700 1 -- 172.21.15.146:0/3953306259 shutdown_connections 2024-01-21T20:07:35.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:35.501+0000 7ff22b16a700 1 --2- 172.21.15.146:0/3953306259 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ff210030000 0x7ff2100324c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:35.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:35.501+0000 7ff22b16a700 1 --2- 172.21.15.146:0/3953306259 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2240f62a0 0x7ff2241064b0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:35.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:35.501+0000 7ff22b16a700 1 -- 172.21.15.146:0/3953306259 >> 172.21.15.146:0/3953306259 conn(0x7ff2240f1aa0 msgr2=0x7ff2240f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:35.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:35.501+0000 7ff22b16a700 1 -- 172.21.15.146:0/3953306259 shutdown_connections 2024-01-21T20:07:35.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:35.501+0000 7ff22b16a700 1 -- 172.21.15.146:0/3953306259 wait complete. 2024-01-21T20:07:35.492 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:07:36.432 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:35 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:07:35.500700+0000 mon.smithi053 (mon.0) 309 : audit [DBG] from='client.? 172.21.15.146:0/3953306259' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:07:36.920 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:07:36.920 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:07:37.433 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:36 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:07:35.206522+0000 mgr.smithi053.fgsmol (mgr.14164) 261 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:38.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:38.127+0000 7f2f697e1700 1 -- 172.21.15.146:0/3731581414 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f640f59d0 msgr2=0x7f2f640f5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:38.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:38.127+0000 7f2f697e1700 1 --2- 172.21.15.146:0/3731581414 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f640f59d0 0x7f2f640f5df0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f2f58009fb0 tx=0x7f2f58009b30).stop 2024-01-21T20:07:38.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:38.127+0000 7f2f697e1700 1 -- 172.21.15.146:0/3731581414 shutdown_connections 2024-01-21T20:07:38.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:38.127+0000 7f2f697e1700 1 --2- 172.21.15.146:0/3731581414 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f640f59d0 0x7f2f640f5df0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:38.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:38.127+0000 7f2f697e1700 1 -- 172.21.15.146:0/3731581414 >> 172.21.15.146:0/3731581414 conn(0x7f2f640f02e0 msgr2=0x7f2f640f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:38.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:38.127+0000 7f2f697e1700 1 -- 172.21.15.146:0/3731581414 shutdown_connections 2024-01-21T20:07:38.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:38.127+0000 7f2f697e1700 1 -- 172.21.15.146:0/3731581414 wait complete. 2024-01-21T20:07:38.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:38.128+0000 7f2f697e1700 1 Processor -- start 2024-01-21T20:07:38.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:38.128+0000 7f2f697e1700 1 -- start start 2024-01-21T20:07:38.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:38.128+0000 7f2f697e1700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f640f59d0 0x7f2f641066c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:38.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:38.128+0000 7f2f697e1700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f64106bb0 con 0x7f2f640f59d0 2024-01-21T20:07:38.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:38.128+0000 7f2f62ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f640f59d0 0x7f2f641066c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:38.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:38.128+0000 7f2f62ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f640f59d0 0x7f2f641066c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:54432/0 (socket says 172.21.15.146:54432) 2024-01-21T20:07:38.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:38.128+0000 7f2f62ffd700 1 -- 172.21.15.146:0/1269378291 learned_addr learned my addr 172.21.15.146:0/1269378291 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:07:38.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:38.129+0000 7f2f62ffd700 1 -- 172.21.15.146:0/1269378291 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f58005040 con 0x7f2f640f59d0 2024-01-21T20:07:38.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:38.129+0000 7f2f62ffd700 1 --2- 172.21.15.146:0/1269378291 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f640f59d0 0x7f2f641066c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f2f58013510 tx=0x7f2f5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:38.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:38.129+0000 7f2f4b7fe700 1 -- 172.21.15.146:0/1269378291 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f58009320 con 0x7f2f640f59d0 2024-01-21T20:07:38.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:38.129+0000 7f2f697e1700 1 -- 172.21.15.146:0/1269378291 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f64106db0 con 0x7f2f640f59d0 2024-01-21T20:07:38.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:38.129+0000 7f2f4b7fe700 1 -- 172.21.15.146:0/1269378291 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2f58013f00 con 0x7f2f640f59d0 2024-01-21T20:07:38.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:38.129+0000 7f2f4b7fe700 1 -- 172.21.15.146:0/1269378291 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f58014d90 con 0x7f2f640f59d0 2024-01-21T20:07:38.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:38.129+0000 7f2f697e1700 1 -- 172.21.15.146:0/1269378291 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f64107240 con 0x7f2f640f59d0 2024-01-21T20:07:38.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:38.130+0000 7f2f4b7fe700 1 -- 172.21.15.146:0/1269378291 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2f580095f0 con 0x7f2f640f59d0 2024-01-21T20:07:38.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:38.130+0000 7f2f697e1700 1 -- 172.21.15.146:0/1269378291 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f64045ad0 con 0x7f2f640f59d0 2024-01-21T20:07:38.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:38.130+0000 7f2f4b7fe700 1 --2- 172.21.15.146:0/1269378291 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f2f4c02ff60 0x7f2f4c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:38.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:38.130+0000 7f2f4b7fe700 1 -- 172.21.15.146:0/1269378291 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2f58040fc0 con 0x7f2f640f59d0 2024-01-21T20:07:38.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:38.130+0000 7f2f627fc700 1 --2- 172.21.15.146:0/1269378291 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f2f4c02ff60 0x7f2f4c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:38.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:38.131+0000 7f2f627fc700 1 --2- 172.21.15.146:0/1269378291 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f2f4c02ff60 0x7f2f4c032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f2f540096a0 tx=0x7f2f54006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:38.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:38.135+0000 7f2f4b7fe700 1 -- 172.21.15.146:0/1269378291 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2f58014420 con 0x7f2f640f59d0 2024-01-21T20:07:38.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:38.340+0000 7f2f697e1700 1 -- 172.21.15.146:0/1269378291 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2f64108090 con 0x7f2f640f59d0 2024-01-21T20:07:38.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:38.340+0000 7f2f4b7fe700 1 -- 172.21.15.146:0/1269378291 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2f58010230 con 0x7f2f640f59d0 2024-01-21T20:07:38.331 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:07:38.331 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:07:38.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:38.342+0000 7f2f697e1700 1 -- 172.21.15.146:0/1269378291 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f2f4c02ff60 msgr2=0x7f2f4c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:38.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:38.342+0000 7f2f697e1700 1 --2- 172.21.15.146:0/1269378291 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f2f4c02ff60 0x7f2f4c032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f2f540096a0 tx=0x7f2f54006b40).stop 2024-01-21T20:07:38.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:38.342+0000 7f2f697e1700 1 -- 172.21.15.146:0/1269378291 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f640f59d0 msgr2=0x7f2f641066c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:38.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:38.342+0000 7f2f697e1700 1 --2- 172.21.15.146:0/1269378291 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f640f59d0 0x7f2f641066c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f2f58013510 tx=0x7f2f5800b040).stop 2024-01-21T20:07:38.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:38.342+0000 7f2f697e1700 1 -- 172.21.15.146:0/1269378291 shutdown_connections 2024-01-21T20:07:38.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:38.342+0000 7f2f697e1700 1 --2- 172.21.15.146:0/1269378291 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f2f4c02ff60 0x7f2f4c032420 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:38.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:38.342+0000 7f2f697e1700 1 --2- 172.21.15.146:0/1269378291 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f640f59d0 0x7f2f641066c0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:38.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:38.342+0000 7f2f697e1700 1 -- 172.21.15.146:0/1269378291 >> 172.21.15.146:0/1269378291 conn(0x7f2f640f02e0 msgr2=0x7f2f640f4f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:38.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:38.342+0000 7f2f697e1700 1 -- 172.21.15.146:0/1269378291 shutdown_connections 2024-01-21T20:07:38.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:38.342+0000 7f2f697e1700 1 -- 172.21.15.146:0/1269378291 wait complete. 2024-01-21T20:07:38.333 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:07:39.433 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:38 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:07:37.206879+0000 mgr.smithi053.fgsmol (mgr.14164) 262 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:39.433 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:38 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:07:38.341651+0000 mon.smithi053 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.146:0/1269378291' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:07:39.762 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:07:39.762 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:07:40.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:40.990+0000 7ffba8ae6700 1 -- 172.21.15.146:0/2315868776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffba40f6510 msgr2=0x7ffba40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:40.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:40.990+0000 7ffba8ae6700 1 --2- 172.21.15.146:0/2315868776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffba40f6510 0x7ffba40f6930 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7ffb94009fb0 tx=0x7ffb94009b30).stop 2024-01-21T20:07:40.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:40.990+0000 7ffba8ae6700 1 -- 172.21.15.146:0/2315868776 shutdown_connections 2024-01-21T20:07:40.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:40.990+0000 7ffba8ae6700 1 --2- 172.21.15.146:0/2315868776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffba40f6510 0x7ffba40f6930 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:40.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:40.990+0000 7ffba8ae6700 1 -- 172.21.15.146:0/2315868776 >> 172.21.15.146:0/2315868776 conn(0x7ffba40f1c70 msgr2=0x7ffba40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:40.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:40.990+0000 7ffba8ae6700 1 -- 172.21.15.146:0/2315868776 shutdown_connections 2024-01-21T20:07:40.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:40.990+0000 7ffba8ae6700 1 -- 172.21.15.146:0/2315868776 wait complete. 2024-01-21T20:07:40.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:40.991+0000 7ffba8ae6700 1 Processor -- start 2024-01-21T20:07:40.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:40.991+0000 7ffba8ae6700 1 -- start start 2024-01-21T20:07:40.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:40.991+0000 7ffba8ae6700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffba40f6510 0x7ffba4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:40.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:40.991+0000 7ffba8ae6700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffba4106c20 con 0x7ffba40f6510 2024-01-21T20:07:40.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:40.991+0000 7ffba259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffba40f6510 0x7ffba4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:40.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:40.991+0000 7ffba259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffba40f6510 0x7ffba4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:54442/0 (socket says 172.21.15.146:54442) 2024-01-21T20:07:40.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:40.991+0000 7ffba259c700 1 -- 172.21.15.146:0/2972291896 learned_addr learned my addr 172.21.15.146:0/2972291896 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:07:40.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:40.992+0000 7ffba259c700 1 -- 172.21.15.146:0/2972291896 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb94005040 con 0x7ffba40f6510 2024-01-21T20:07:40.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:40.992+0000 7ffba259c700 1 --2- 172.21.15.146:0/2972291896 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffba40f6510 0x7ffba4106730 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7ffb94013510 tx=0x7ffb9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:40.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:40.992+0000 7ffb9affd700 1 -- 172.21.15.146:0/2972291896 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb94009320 con 0x7ffba40f6510 2024-01-21T20:07:40.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:40.992+0000 7ffb9affd700 1 -- 172.21.15.146:0/2972291896 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffb94013f00 con 0x7ffba40f6510 2024-01-21T20:07:40.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:40.992+0000 7ffba8ae6700 1 -- 172.21.15.146:0/2972291896 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffba4106e20 con 0x7ffba40f6510 2024-01-21T20:07:40.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:40.992+0000 7ffb9affd700 1 -- 172.21.15.146:0/2972291896 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb94014d90 con 0x7ffba40f6510 2024-01-21T20:07:40.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:40.992+0000 7ffba8ae6700 1 -- 172.21.15.146:0/2972291896 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffba41072b0 con 0x7ffba40f6510 2024-01-21T20:07:40.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:40.993+0000 7ffb9affd700 1 -- 172.21.15.146:0/2972291896 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffb940095f0 con 0x7ffba40f6510 2024-01-21T20:07:40.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:40.993+0000 7ffba8ae6700 1 -- 172.21.15.146:0/2972291896 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffba4045ad0 con 0x7ffba40f6510 2024-01-21T20:07:40.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:40.993+0000 7ffb9affd700 1 --2- 172.21.15.146:0/2972291896 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ffb9002ff60 0x7ffb90032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:40.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:40.993+0000 7ffb9affd700 1 -- 172.21.15.146:0/2972291896 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffb94041000 con 0x7ffba40f6510 2024-01-21T20:07:40.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:40.993+0000 7ffba1d9b700 1 --2- 172.21.15.146:0/2972291896 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ffb9002ff60 0x7ffb90032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:40.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:40.993+0000 7ffba1d9b700 1 --2- 172.21.15.146:0/2972291896 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ffb9002ff60 0x7ffb90032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7ffb8c0096a0 tx=0x7ffb8c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:40.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:40.998+0000 7ffb9affd700 1 -- 172.21.15.146:0/2972291896 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffb94014420 con 0x7ffba40f6510 2024-01-21T20:07:41.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:41.201+0000 7ffba8ae6700 1 -- 172.21.15.146:0/2972291896 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffba40fbf60 con 0x7ffba40f6510 2024-01-21T20:07:41.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:41.202+0000 7ffb9affd700 1 -- 172.21.15.146:0/2972291896 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ffb9401d7f0 con 0x7ffba40f6510 2024-01-21T20:07:41.192 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:07:41.192 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:07:41.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:41.203+0000 7ffba8ae6700 1 -- 172.21.15.146:0/2972291896 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ffb9002ff60 msgr2=0x7ffb90032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:41.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:41.203+0000 7ffba8ae6700 1 --2- 172.21.15.146:0/2972291896 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ffb9002ff60 0x7ffb90032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7ffb8c0096a0 tx=0x7ffb8c006b40).stop 2024-01-21T20:07:41.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:41.203+0000 7ffba8ae6700 1 -- 172.21.15.146:0/2972291896 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffba40f6510 msgr2=0x7ffba4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:41.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:41.203+0000 7ffba8ae6700 1 --2- 172.21.15.146:0/2972291896 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffba40f6510 0x7ffba4106730 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7ffb94013510 tx=0x7ffb9400b040).stop 2024-01-21T20:07:41.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:41.203+0000 7ffba8ae6700 1 -- 172.21.15.146:0/2972291896 shutdown_connections 2024-01-21T20:07:41.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:41.203+0000 7ffba8ae6700 1 --2- 172.21.15.146:0/2972291896 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7ffb9002ff60 0x7ffb90032420 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:41.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:41.203+0000 7ffba8ae6700 1 --2- 172.21.15.146:0/2972291896 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffba40f6510 0x7ffba4106730 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:41.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:41.203+0000 7ffba8ae6700 1 -- 172.21.15.146:0/2972291896 >> 172.21.15.146:0/2972291896 conn(0x7ffba40f1c70 msgr2=0x7ffba40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:41.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:41.203+0000 7ffba8ae6700 1 -- 172.21.15.146:0/2972291896 shutdown_connections 2024-01-21T20:07:41.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:41.204+0000 7ffba8ae6700 1 -- 172.21.15.146:0/2972291896 wait complete. 2024-01-21T20:07:41.195 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:07:41.432 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:40 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:07:39.207207+0000 mgr.smithi053.fgsmol (mgr.14164) 263 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:42.432 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:42 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:07:41.203295+0000 mon.smithi053 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.146:0/2972291896' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:07:42.617 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:07:42.618 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:07:43.432 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:43 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:07:41.207439+0000 mgr.smithi053.fgsmol (mgr.14164) 264 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:43.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:43.883+0000 7fdb0234f700 1 -- 172.21.15.146:0/1363727651 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdafc060280 msgr2=0x7fdafc0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:43.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:43.883+0000 7fdb0234f700 1 --2- 172.21.15.146:0/1363727651 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdafc060280 0x7fdafc0f90f0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7fdaec009fb0 tx=0x7fdaec009b30).stop 2024-01-21T20:07:43.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:43.883+0000 7fdb0234f700 1 -- 172.21.15.146:0/1363727651 shutdown_connections 2024-01-21T20:07:43.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:43.883+0000 7fdb0234f700 1 --2- 172.21.15.146:0/1363727651 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdafc060280 0x7fdafc0f90f0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:43.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:43.883+0000 7fdb0234f700 1 -- 172.21.15.146:0/1363727651 >> 172.21.15.146:0/1363727651 conn(0x7fdafc0f0b70 msgr2=0x7fdafc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:43.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:43.883+0000 7fdb0234f700 1 -- 172.21.15.146:0/1363727651 shutdown_connections 2024-01-21T20:07:43.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:43.883+0000 7fdb0234f700 1 -- 172.21.15.146:0/1363727651 wait complete. 2024-01-21T20:07:43.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:43.883+0000 7fdb0234f700 1 Processor -- start 2024-01-21T20:07:43.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:43.883+0000 7fdb0234f700 1 -- start start 2024-01-21T20:07:43.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:43.883+0000 7fdb0234f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdafc060280 0x7fdafc1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:43.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:43.883+0000 7fdb0234f700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdafc106bd0 con 0x7fdafc060280 2024-01-21T20:07:43.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:43.884+0000 7fdafbfff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdafc060280 0x7fdafc1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:43.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:43.884+0000 7fdafbfff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdafc060280 0x7fdafc1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:54462/0 (socket says 172.21.15.146:54462) 2024-01-21T20:07:43.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:43.884+0000 7fdafbfff700 1 -- 172.21.15.146:0/3310681740 learned_addr learned my addr 172.21.15.146:0/3310681740 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:07:43.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:43.884+0000 7fdafbfff700 1 -- 172.21.15.146:0/3310681740 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaec005040 con 0x7fdafc060280 2024-01-21T20:07:43.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:43.884+0000 7fdafbfff700 1 --2- 172.21.15.146:0/3310681740 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdafc060280 0x7fdafc1066e0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fdaec013510 tx=0x7fdaec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:43.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:43.884+0000 7fdaf8ff9700 1 -- 172.21.15.146:0/3310681740 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdaec009320 con 0x7fdafc060280 2024-01-21T20:07:43.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:43.884+0000 7fdb0234f700 1 -- 172.21.15.146:0/3310681740 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdafc106dd0 con 0x7fdafc060280 2024-01-21T20:07:43.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:43.884+0000 7fdaf8ff9700 1 -- 172.21.15.146:0/3310681740 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdaec013f00 con 0x7fdafc060280 2024-01-21T20:07:43.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:43.884+0000 7fdaf8ff9700 1 -- 172.21.15.146:0/3310681740 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdaec014dc0 con 0x7fdafc060280 2024-01-21T20:07:43.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:43.884+0000 7fdb0234f700 1 -- 172.21.15.146:0/3310681740 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdafc107260 con 0x7fdafc060280 2024-01-21T20:07:43.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:43.885+0000 7fdaf8ff9700 1 -- 172.21.15.146:0/3310681740 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdaec014420 con 0x7fdafc060280 2024-01-21T20:07:43.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:43.885+0000 7fdb0234f700 1 -- 172.21.15.146:0/3310681740 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdafc045ad0 con 0x7fdafc060280 2024-01-21T20:07:43.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:43.885+0000 7fdaf8ff9700 1 --2- 172.21.15.146:0/3310681740 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fdae402ff60 0x7fdae4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:43.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:43.885+0000 7fdaf8ff9700 1 -- 172.21.15.146:0/3310681740 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdaec041110 con 0x7fdafc060280 2024-01-21T20:07:43.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:43.885+0000 7fdafb7fe700 1 --2- 172.21.15.146:0/3310681740 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fdae402ff60 0x7fdae4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:43.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:43.886+0000 7fdafb7fe700 1 --2- 172.21.15.146:0/3310681740 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fdae402ff60 0x7fdae4032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fdaf00096a0 tx=0x7fdaf0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:43.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:43.890+0000 7fdaf8ff9700 1 -- 172.21.15.146:0/3310681740 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdaec0095f0 con 0x7fdafc060280 2024-01-21T20:07:44.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:44.094+0000 7fdb0234f700 1 -- 172.21.15.146:0/3310681740 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdafc108110 con 0x7fdafc060280 2024-01-21T20:07:44.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:44.095+0000 7fdaf8ff9700 1 -- 172.21.15.146:0/3310681740 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdaec01cb10 con 0x7fdafc060280 2024-01-21T20:07:44.085 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:07:44.085 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:07:44.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:44.096+0000 7fdb0234f700 1 -- 172.21.15.146:0/3310681740 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fdae402ff60 msgr2=0x7fdae4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:44.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:44.096+0000 7fdb0234f700 1 --2- 172.21.15.146:0/3310681740 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fdae402ff60 0x7fdae4032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fdaf00096a0 tx=0x7fdaf0006b40).stop 2024-01-21T20:07:44.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:44.096+0000 7fdb0234f700 1 -- 172.21.15.146:0/3310681740 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdafc060280 msgr2=0x7fdafc1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:44.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:44.096+0000 7fdb0234f700 1 --2- 172.21.15.146:0/3310681740 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdafc060280 0x7fdafc1066e0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fdaec013510 tx=0x7fdaec00b040).stop 2024-01-21T20:07:44.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:44.096+0000 7fdb0234f700 1 -- 172.21.15.146:0/3310681740 shutdown_connections 2024-01-21T20:07:44.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:44.096+0000 7fdb0234f700 1 --2- 172.21.15.146:0/3310681740 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fdae402ff60 0x7fdae4032420 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:44.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:44.096+0000 7fdb0234f700 1 --2- 172.21.15.146:0/3310681740 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdafc060280 0x7fdafc1066e0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:44.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:44.096+0000 7fdb0234f700 1 -- 172.21.15.146:0/3310681740 >> 172.21.15.146:0/3310681740 conn(0x7fdafc0f0b70 msgr2=0x7fdafc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:44.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:44.097+0000 7fdb0234f700 1 -- 172.21.15.146:0/3310681740 shutdown_connections 2024-01-21T20:07:44.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:44.097+0000 7fdb0234f700 1 -- 172.21.15.146:0/3310681740 wait complete. 2024-01-21T20:07:44.087 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:07:44.432 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:44 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:07:43.207794+0000 mgr.smithi053.fgsmol (mgr.14164) 265 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:45.432 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:45 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:07:44.095388+0000 mon.smithi053 (mon.0) 312 : audit [DBG] from='client.? 172.21.15.146:0/3310681740' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:07:45.516 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:07:45.516 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:07:46.432 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:46 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:07:45.208161+0000 mgr.smithi053.fgsmol (mgr.14164) 266 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:46.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:46.706+0000 7f24d796a700 1 -- 172.21.15.146:0/1982119776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24d00f6510 msgr2=0x7f24d00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:46.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:46.706+0000 7f24d796a700 1 --2- 172.21.15.146:0/1982119776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24d00f6510 0x7f24d00f6930 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f24c0009fb0 tx=0x7f24c0009b30).stop 2024-01-21T20:07:46.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:46.706+0000 7f24d796a700 1 -- 172.21.15.146:0/1982119776 shutdown_connections 2024-01-21T20:07:46.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:46.706+0000 7f24d796a700 1 --2- 172.21.15.146:0/1982119776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24d00f6510 0x7f24d00f6930 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:46.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:46.706+0000 7f24d796a700 1 -- 172.21.15.146:0/1982119776 >> 172.21.15.146:0/1982119776 conn(0x7f24d00f1c70 msgr2=0x7f24d00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:46.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:46.706+0000 7f24d796a700 1 -- 172.21.15.146:0/1982119776 shutdown_connections 2024-01-21T20:07:46.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:46.706+0000 7f24d796a700 1 -- 172.21.15.146:0/1982119776 wait complete. 2024-01-21T20:07:46.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:46.707+0000 7f24d796a700 1 Processor -- start 2024-01-21T20:07:46.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:46.707+0000 7f24d796a700 1 -- start start 2024-01-21T20:07:46.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:46.707+0000 7f24d796a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24d00f6510 0x7f24d0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:46.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:46.707+0000 7f24d796a700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24d0106c20 con 0x7f24d00f6510 2024-01-21T20:07:46.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:46.707+0000 7f24d5706700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24d00f6510 0x7f24d0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:46.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:46.707+0000 7f24d5706700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24d00f6510 0x7f24d0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:54484/0 (socket says 172.21.15.146:54484) 2024-01-21T20:07:46.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:46.707+0000 7f24d5706700 1 -- 172.21.15.146:0/4191620305 learned_addr learned my addr 172.21.15.146:0/4191620305 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:07:46.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:46.708+0000 7f24d5706700 1 -- 172.21.15.146:0/4191620305 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24c0005040 con 0x7f24d00f6510 2024-01-21T20:07:46.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:46.708+0000 7f24d5706700 1 --2- 172.21.15.146:0/4191620305 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24d00f6510 0x7f24d0106730 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f24c0013510 tx=0x7f24c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:46.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:46.708+0000 7f24c5ffb700 1 -- 172.21.15.146:0/4191620305 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f24c0009320 con 0x7f24d00f6510 2024-01-21T20:07:46.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:46.708+0000 7f24d796a700 1 -- 172.21.15.146:0/4191620305 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24d0106e20 con 0x7f24d00f6510 2024-01-21T20:07:46.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:46.708+0000 7f24c5ffb700 1 -- 172.21.15.146:0/4191620305 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f24c0013f00 con 0x7f24d00f6510 2024-01-21T20:07:46.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:46.708+0000 7f24c5ffb700 1 -- 172.21.15.146:0/4191620305 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f24c0014d90 con 0x7f24d00f6510 2024-01-21T20:07:46.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:46.708+0000 7f24d796a700 1 -- 172.21.15.146:0/4191620305 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24d01072b0 con 0x7f24d00f6510 2024-01-21T20:07:46.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:46.709+0000 7f24c5ffb700 1 -- 172.21.15.146:0/4191620305 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f24c00263a0 con 0x7f24d00f6510 2024-01-21T20:07:46.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:46.709+0000 7f24c5ffb700 1 --2- 172.21.15.146:0/4191620305 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f24bc02ff60 0x7f24bc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:46.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:46.709+0000 7f24c5ffb700 1 -- 172.21.15.146:0/4191620305 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f24c00410b0 con 0x7f24d00f6510 2024-01-21T20:07:46.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:46.709+0000 7f24d796a700 1 -- 172.21.15.146:0/4191620305 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24d0045ad0 con 0x7f24d00f6510 2024-01-21T20:07:46.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:46.709+0000 7f24d4f05700 1 --2- 172.21.15.146:0/4191620305 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f24bc02ff60 0x7f24bc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:46.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:46.710+0000 7f24d4f05700 1 --2- 172.21.15.146:0/4191620305 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f24bc02ff60 0x7f24bc032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f24cc0096a0 tx=0x7f24cc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:46.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:46.714+0000 7f24c5ffb700 1 -- 172.21.15.146:0/4191620305 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f24c0012920 con 0x7f24d00f6510 2024-01-21T20:07:46.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:46.919+0000 7f24d796a700 1 -- 172.21.15.146:0/4191620305 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f24d00fc050 con 0x7f24d00f6510 2024-01-21T20:07:46.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:46.919+0000 7f24c5ffb700 1 -- 172.21.15.146:0/4191620305 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f24c002b020 con 0x7f24d00f6510 2024-01-21T20:07:46.910 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:07:46.910 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:07:46.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:46.921+0000 7f24d796a700 1 -- 172.21.15.146:0/4191620305 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f24bc02ff60 msgr2=0x7f24bc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:46.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:46.921+0000 7f24d796a700 1 --2- 172.21.15.146:0/4191620305 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f24bc02ff60 0x7f24bc032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f24cc0096a0 tx=0x7f24cc006b40).stop 2024-01-21T20:07:46.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:46.921+0000 7f24d796a700 1 -- 172.21.15.146:0/4191620305 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24d00f6510 msgr2=0x7f24d0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:46.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:46.921+0000 7f24d796a700 1 --2- 172.21.15.146:0/4191620305 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24d00f6510 0x7f24d0106730 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f24c0013510 tx=0x7f24c000b040).stop 2024-01-21T20:07:46.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:46.921+0000 7f24d796a700 1 -- 172.21.15.146:0/4191620305 shutdown_connections 2024-01-21T20:07:46.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:46.921+0000 7f24d796a700 1 --2- 172.21.15.146:0/4191620305 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f24bc02ff60 0x7f24bc032420 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:46.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:46.921+0000 7f24d796a700 1 --2- 172.21.15.146:0/4191620305 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24d00f6510 0x7f24d0106730 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:46.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:46.921+0000 7f24d796a700 1 -- 172.21.15.146:0/4191620305 >> 172.21.15.146:0/4191620305 conn(0x7f24d00f1c70 msgr2=0x7f24d00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:46.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:46.921+0000 7f24d796a700 1 -- 172.21.15.146:0/4191620305 shutdown_connections 2024-01-21T20:07:46.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:46.921+0000 7f24d796a700 1 -- 172.21.15.146:0/4191620305 wait complete. 2024-01-21T20:07:46.912 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:07:47.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:47 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:07:46.919658+0000 mon.smithi053 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.146:0/4191620305' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:07:48.327 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:07:48.327 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:07:48.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:48 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:07:47.208547+0000 mgr.smithi053.fgsmol (mgr.14164) 267 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:49.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:49.597+0000 7f58f7151700 1 -- 172.21.15.146:0/2272862088 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58f0060280 msgr2=0x7f58f00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:49.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:49.597+0000 7f58f7151700 1 --2- 172.21.15.146:0/2272862088 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58f0060280 0x7f58f00f90f0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f58e4009fb0 tx=0x7f58e4009b30).stop 2024-01-21T20:07:49.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:49.597+0000 7f58f7151700 1 -- 172.21.15.146:0/2272862088 shutdown_connections 2024-01-21T20:07:49.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:49.597+0000 7f58f7151700 1 --2- 172.21.15.146:0/2272862088 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58f0060280 0x7f58f00f90f0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:49.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:49.597+0000 7f58f7151700 1 -- 172.21.15.146:0/2272862088 >> 172.21.15.146:0/2272862088 conn(0x7f58f00f0b70 msgr2=0x7f58f00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:49.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:49.597+0000 7f58f7151700 1 -- 172.21.15.146:0/2272862088 shutdown_connections 2024-01-21T20:07:49.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:49.597+0000 7f58f7151700 1 -- 172.21.15.146:0/2272862088 wait complete. 2024-01-21T20:07:49.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:49.597+0000 7f58f7151700 1 Processor -- start 2024-01-21T20:07:49.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:49.597+0000 7f58f7151700 1 -- start start 2024-01-21T20:07:49.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:49.598+0000 7f58f7151700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58f0060280 0x7f58f01066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:49.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:49.598+0000 7f58f7151700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f0106bd0 con 0x7f58f0060280 2024-01-21T20:07:49.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:49.598+0000 7f58f4eed700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58f0060280 0x7f58f01066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:49.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:49.598+0000 7f58f4eed700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58f0060280 0x7f58f01066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:33810/0 (socket says 172.21.15.146:33810) 2024-01-21T20:07:49.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:49.598+0000 7f58f4eed700 1 -- 172.21.15.146:0/88691071 learned_addr learned my addr 172.21.15.146:0/88691071 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:07:49.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:49.598+0000 7f58f4eed700 1 -- 172.21.15.146:0/88691071 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e4005040 con 0x7f58f0060280 2024-01-21T20:07:49.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:49.598+0000 7f58f4eed700 1 --2- 172.21.15.146:0/88691071 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58f0060280 0x7f58f01066e0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f58e4009830 tx=0x7f58e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:49.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:49.598+0000 7f58ed7fa700 1 -- 172.21.15.146:0/88691071 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58e4009320 con 0x7f58f0060280 2024-01-21T20:07:49.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:49.598+0000 7f58ed7fa700 1 -- 172.21.15.146:0/88691071 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f58e40136c0 con 0x7f58f0060280 2024-01-21T20:07:49.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:49.598+0000 7f58f7151700 1 -- 172.21.15.146:0/88691071 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58f0106dd0 con 0x7f58f0060280 2024-01-21T20:07:49.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:49.598+0000 7f58ed7fa700 1 -- 172.21.15.146:0/88691071 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58e401ee90 con 0x7f58f0060280 2024-01-21T20:07:49.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:49.598+0000 7f58f7151700 1 -- 172.21.15.146:0/88691071 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58f0107260 con 0x7f58f0060280 2024-01-21T20:07:49.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:49.599+0000 7f58ed7fa700 1 -- 172.21.15.146:0/88691071 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f58e401e420 con 0x7f58f0060280 2024-01-21T20:07:49.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:49.599+0000 7f58f7151700 1 -- 172.21.15.146:0/88691071 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58f0045ad0 con 0x7f58f0060280 2024-01-21T20:07:49.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:49.599+0000 7f58ed7fa700 1 --2- 172.21.15.146:0/88691071 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f58e002ff10 0x7f58e00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:49.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:49.599+0000 7f58ed7fa700 1 -- 172.21.15.146:0/88691071 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f58e4040fc0 con 0x7f58f0060280 2024-01-21T20:07:49.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:49.600+0000 7f58effff700 1 --2- 172.21.15.146:0/88691071 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f58e002ff10 0x7f58e00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:49.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:49.600+0000 7f58effff700 1 --2- 172.21.15.146:0/88691071 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f58e002ff10 0x7f58e00323d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f58dc0096a0 tx=0x7f58dc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:49.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:49.604+0000 7f58ed7fa700 1 -- 172.21.15.146:0/88691071 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f58e406d2b0 con 0x7f58f0060280 2024-01-21T20:07:49.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:49.809+0000 7f58f7151700 1 -- 172.21.15.146:0/88691071 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f58f0108110 con 0x7f58f0060280 2024-01-21T20:07:49.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:49.810+0000 7f58ed7fa700 1 -- 172.21.15.146:0/88691071 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f58e40095f0 con 0x7f58f0060280 2024-01-21T20:07:49.800 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:07:49.800 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:07:49.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:49.811+0000 7f58f7151700 1 -- 172.21.15.146:0/88691071 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f58e002ff10 msgr2=0x7f58e00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:49.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:49.811+0000 7f58f7151700 1 --2- 172.21.15.146:0/88691071 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f58e002ff10 0x7f58e00323d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f58dc0096a0 tx=0x7f58dc006b40).stop 2024-01-21T20:07:49.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:49.811+0000 7f58f7151700 1 -- 172.21.15.146:0/88691071 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58f0060280 msgr2=0x7f58f01066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:49.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:49.811+0000 7f58f7151700 1 --2- 172.21.15.146:0/88691071 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58f0060280 0x7f58f01066e0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f58e4009830 tx=0x7f58e400b040).stop 2024-01-21T20:07:49.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:49.811+0000 7f58f7151700 1 -- 172.21.15.146:0/88691071 shutdown_connections 2024-01-21T20:07:49.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:49.811+0000 7f58f7151700 1 --2- 172.21.15.146:0/88691071 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f58e002ff10 0x7f58e00323d0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:49.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:49.811+0000 7f58f7151700 1 --2- 172.21.15.146:0/88691071 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58f0060280 0x7f58f01066e0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:49.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:49.811+0000 7f58f7151700 1 -- 172.21.15.146:0/88691071 >> 172.21.15.146:0/88691071 conn(0x7f58f00f0b70 msgr2=0x7f58f00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:49.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:49.812+0000 7f58f7151700 1 -- 172.21.15.146:0/88691071 shutdown_connections 2024-01-21T20:07:49.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:49.812+0000 7f58f7151700 1 -- 172.21.15.146:0/88691071 wait complete. 2024-01-21T20:07:49.802 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:07:50.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:50 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:07:49.208965+0000 mgr.smithi053.fgsmol (mgr.14164) 268 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:50.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:50 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:07:49.810384+0000 mon.smithi053 (mon.0) 314 : audit [DBG] from='client.? 172.21.15.146:0/88691071' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:07:51.233 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:07:51.233 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:07:52.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:52.477+0000 7f4689c1c700 1 -- 172.21.15.146:0/4185864105 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46840f6510 msgr2=0x7f46840f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:52.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:52.477+0000 7f4689c1c700 1 --2- 172.21.15.146:0/4185864105 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46840f6510 0x7f46840f6930 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f466c009fb0 tx=0x7f466c009b30).stop 2024-01-21T20:07:52.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:52.477+0000 7f4689c1c700 1 -- 172.21.15.146:0/4185864105 shutdown_connections 2024-01-21T20:07:52.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:52.477+0000 7f4689c1c700 1 --2- 172.21.15.146:0/4185864105 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46840f6510 0x7f46840f6930 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:52.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:52.477+0000 7f4689c1c700 1 -- 172.21.15.146:0/4185864105 >> 172.21.15.146:0/4185864105 conn(0x7f46840f1c70 msgr2=0x7f46840f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:52.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:52.477+0000 7f4689c1c700 1 -- 172.21.15.146:0/4185864105 shutdown_connections 2024-01-21T20:07:52.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:52.477+0000 7f4689c1c700 1 -- 172.21.15.146:0/4185864105 wait complete. 2024-01-21T20:07:52.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:52.478+0000 7f4689c1c700 1 Processor -- start 2024-01-21T20:07:52.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:52.478+0000 7f4689c1c700 1 -- start start 2024-01-21T20:07:52.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:52.478+0000 7f4689c1c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46840f6510 0x7f4684106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:52.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:52.478+0000 7f4689c1c700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4684106c20 con 0x7f46840f6510 2024-01-21T20:07:52.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:52.479+0000 7f46837fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46840f6510 0x7f4684106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:52.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:52.479+0000 7f46837fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46840f6510 0x7f4684106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:33830/0 (socket says 172.21.15.146:33830) 2024-01-21T20:07:52.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:52.479+0000 7f46837fe700 1 -- 172.21.15.146:0/3652285804 learned_addr learned my addr 172.21.15.146:0/3652285804 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:07:52.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:52.479+0000 7f46837fe700 1 -- 172.21.15.146:0/3652285804 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f466c005040 con 0x7f46840f6510 2024-01-21T20:07:52.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:52.479+0000 7f46837fe700 1 --2- 172.21.15.146:0/3652285804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46840f6510 0x7f4684106730 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f466c013510 tx=0x7f466c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:52.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:52.479+0000 7f467bfff700 1 -- 172.21.15.146:0/3652285804 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f466c009320 con 0x7f46840f6510 2024-01-21T20:07:52.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:52.479+0000 7f4689c1c700 1 -- 172.21.15.146:0/3652285804 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4684106e20 con 0x7f46840f6510 2024-01-21T20:07:52.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:52.479+0000 7f467bfff700 1 -- 172.21.15.146:0/3652285804 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f466c013f00 con 0x7f46840f6510 2024-01-21T20:07:52.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:52.479+0000 7f467bfff700 1 -- 172.21.15.146:0/3652285804 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f466c014d90 con 0x7f46840f6510 2024-01-21T20:07:52.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:52.479+0000 7f4689c1c700 1 -- 172.21.15.146:0/3652285804 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46841072b0 con 0x7f46840f6510 2024-01-21T20:07:52.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:52.480+0000 7f467bfff700 1 -- 172.21.15.146:0/3652285804 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f466c0263a0 con 0x7f46840f6510 2024-01-21T20:07:52.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:52.480+0000 7f467bfff700 1 --2- 172.21.15.146:0/3652285804 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f467002ffb0 0x7f4670032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:52.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:52.480+0000 7f4689c1c700 1 -- 172.21.15.146:0/3652285804 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4684045ad0 con 0x7f46840f6510 2024-01-21T20:07:52.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:52.480+0000 7f467bfff700 1 -- 172.21.15.146:0/3652285804 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f466c0410b0 con 0x7f46840f6510 2024-01-21T20:07:52.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:52.481+0000 7f4682ffd700 1 --2- 172.21.15.146:0/3652285804 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f467002ffb0 0x7f4670032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:52.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:52.481+0000 7f4682ffd700 1 --2- 172.21.15.146:0/3652285804 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f467002ffb0 0x7f4670032470 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f46740096a0 tx=0x7f4674006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:52.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:52.485+0000 7f467bfff700 1 -- 172.21.15.146:0/3652285804 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f466c012920 con 0x7f46840f6510 2024-01-21T20:07:52.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:52.691+0000 7f4689c1c700 1 -- 172.21.15.146:0/3652285804 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f46840fc050 con 0x7f46840f6510 2024-01-21T20:07:52.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:52.692+0000 7f467bfff700 1 -- 172.21.15.146:0/3652285804 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f466c02b020 con 0x7f46840f6510 2024-01-21T20:07:52.682 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:07:52.682 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:07:52.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:52.694+0000 7f4689c1c700 1 -- 172.21.15.146:0/3652285804 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f467002ffb0 msgr2=0x7f4670032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:52.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:52.694+0000 7f4689c1c700 1 --2- 172.21.15.146:0/3652285804 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f467002ffb0 0x7f4670032470 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f46740096a0 tx=0x7f4674006b40).stop 2024-01-21T20:07:52.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:52.694+0000 7f4689c1c700 1 -- 172.21.15.146:0/3652285804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46840f6510 msgr2=0x7f4684106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:52.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:52.694+0000 7f4689c1c700 1 --2- 172.21.15.146:0/3652285804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46840f6510 0x7f4684106730 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f466c013510 tx=0x7f466c00b040).stop 2024-01-21T20:07:52.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:52.694+0000 7f4689c1c700 1 -- 172.21.15.146:0/3652285804 shutdown_connections 2024-01-21T20:07:52.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:52.694+0000 7f4689c1c700 1 --2- 172.21.15.146:0/3652285804 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f467002ffb0 0x7f4670032470 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:52.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:52.694+0000 7f4689c1c700 1 --2- 172.21.15.146:0/3652285804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46840f6510 0x7f4684106730 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:52.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:52.694+0000 7f4689c1c700 1 -- 172.21.15.146:0/3652285804 >> 172.21.15.146:0/3652285804 conn(0x7f46840f1c70 msgr2=0x7f46840f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:52.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:52.694+0000 7f4689c1c700 1 -- 172.21.15.146:0/3652285804 shutdown_connections 2024-01-21T20:07:52.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:52.694+0000 7f4689c1c700 1 -- 172.21.15.146:0/3652285804 wait complete. 2024-01-21T20:07:52.684 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:07:52.932 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:52 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:07:51.209372+0000 mgr.smithi053.fgsmol (mgr.14164) 269 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:53.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:53 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:07:52.692302+0000 mon.smithi053 (mon.0) 315 : audit [DBG] from='client.? 172.21.15.146:0/3652285804' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:07:54.115 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:07:54.115 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:07:54.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:54 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:07:53.209667+0000 mgr.smithi053.fgsmol (mgr.14164) 270 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:55.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:55.330+0000 7f488f0e6700 1 -- 172.21.15.146:0/3957665222 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48880f5410 msgr2=0x7f48880f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:55.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:55.330+0000 7f488f0e6700 1 --2- 172.21.15.146:0/3957665222 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48880f5410 0x7f48880f5830 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f487c009fb0 tx=0x7f487c009b30).stop 2024-01-21T20:07:55.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:55.330+0000 7f488f0e6700 1 -- 172.21.15.146:0/3957665222 shutdown_connections 2024-01-21T20:07:55.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:55.330+0000 7f488f0e6700 1 --2- 172.21.15.146:0/3957665222 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48880f5410 0x7f48880f5830 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:55.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:55.330+0000 7f488f0e6700 1 -- 172.21.15.146:0/3957665222 >> 172.21.15.146:0/3957665222 conn(0x7f48880f0b70 msgr2=0x7f48880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:55.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:55.330+0000 7f488f0e6700 1 -- 172.21.15.146:0/3957665222 shutdown_connections 2024-01-21T20:07:55.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:55.330+0000 7f488f0e6700 1 -- 172.21.15.146:0/3957665222 wait complete. 2024-01-21T20:07:55.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:55.331+0000 7f488f0e6700 1 Processor -- start 2024-01-21T20:07:55.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:55.331+0000 7f488f0e6700 1 -- start start 2024-01-21T20:07:55.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:55.331+0000 7f488f0e6700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48880f5410 0x7f4888109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:55.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:55.331+0000 7f488f0e6700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4888060240 con 0x7f48880f5410 2024-01-21T20:07:55.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:55.332+0000 7f488ce82700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48880f5410 0x7f4888109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:55.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:55.332+0000 7f488ce82700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48880f5410 0x7f4888109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:33850/0 (socket says 172.21.15.146:33850) 2024-01-21T20:07:55.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:55.332+0000 7f488ce82700 1 -- 172.21.15.146:0/2548990974 learned_addr learned my addr 172.21.15.146:0/2548990974 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:07:55.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:55.332+0000 7f488ce82700 1 -- 172.21.15.146:0/2548990974 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f487c005040 con 0x7f48880f5410 2024-01-21T20:07:55.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:55.332+0000 7f488ce82700 1 --2- 172.21.15.146:0/2548990974 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48880f5410 0x7f4888109430 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f487c013510 tx=0x7f487c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:55.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:55.332+0000 7f48857fa700 1 -- 172.21.15.146:0/2548990974 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f487c009320 con 0x7f48880f5410 2024-01-21T20:07:55.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:55.332+0000 7f488f0e6700 1 -- 172.21.15.146:0/2548990974 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4888060440 con 0x7f48880f5410 2024-01-21T20:07:55.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:55.332+0000 7f48857fa700 1 -- 172.21.15.146:0/2548990974 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f487c013f00 con 0x7f48880f5410 2024-01-21T20:07:55.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:55.332+0000 7f48857fa700 1 -- 172.21.15.146:0/2548990974 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f487c014d90 con 0x7f48880f5410 2024-01-21T20:07:55.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:55.332+0000 7f488f0e6700 1 -- 172.21.15.146:0/2548990974 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4888109ae0 con 0x7f48880f5410 2024-01-21T20:07:55.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:55.333+0000 7f48857fa700 1 -- 172.21.15.146:0/2548990974 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f487c0095f0 con 0x7f48880f5410 2024-01-21T20:07:55.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:55.333+0000 7f488f0e6700 1 -- 172.21.15.146:0/2548990974 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4888045ad0 con 0x7f48880f5410 2024-01-21T20:07:55.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:55.333+0000 7f48857fa700 1 --2- 172.21.15.146:0/2548990974 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f487802ff60 0x7f4878032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:55.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:55.333+0000 7f48857fa700 1 -- 172.21.15.146:0/2548990974 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f487c041080 con 0x7f48880f5410 2024-01-21T20:07:55.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:55.333+0000 7f4887fff700 1 --2- 172.21.15.146:0/2548990974 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f487802ff60 0x7f4878032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:55.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:55.334+0000 7f4887fff700 1 --2- 172.21.15.146:0/2548990974 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f487802ff60 0x7f4878032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f48740096a0 tx=0x7f4874006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:55.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:55.338+0000 7f48857fa700 1 -- 172.21.15.146:0/2548990974 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f487c014420 con 0x7f48880f5410 2024-01-21T20:07:55.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:55.543+0000 7f488f0e6700 1 -- 172.21.15.146:0/2548990974 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f48880fae60 con 0x7f48880f5410 2024-01-21T20:07:55.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:55.544+0000 7f48857fa700 1 -- 172.21.15.146:0/2548990974 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f487c010230 con 0x7f48880f5410 2024-01-21T20:07:55.535 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:07:55.535 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:07:55.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:55.545+0000 7f488f0e6700 1 -- 172.21.15.146:0/2548990974 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f487802ff60 msgr2=0x7f4878032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:55.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:55.545+0000 7f488f0e6700 1 --2- 172.21.15.146:0/2548990974 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f487802ff60 0x7f4878032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f48740096a0 tx=0x7f4874006b40).stop 2024-01-21T20:07:55.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:55.545+0000 7f488f0e6700 1 -- 172.21.15.146:0/2548990974 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48880f5410 msgr2=0x7f4888109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:55.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:55.545+0000 7f488f0e6700 1 --2- 172.21.15.146:0/2548990974 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48880f5410 0x7f4888109430 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f487c013510 tx=0x7f487c00b040).stop 2024-01-21T20:07:55.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:55.545+0000 7f488f0e6700 1 -- 172.21.15.146:0/2548990974 shutdown_connections 2024-01-21T20:07:55.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:55.545+0000 7f488f0e6700 1 --2- 172.21.15.146:0/2548990974 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f487802ff60 0x7f4878032420 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:55.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:55.546+0000 7f488f0e6700 1 --2- 172.21.15.146:0/2548990974 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48880f5410 0x7f4888109430 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:55.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:55.546+0000 7f488f0e6700 1 -- 172.21.15.146:0/2548990974 >> 172.21.15.146:0/2548990974 conn(0x7f48880f0b70 msgr2=0x7f48880f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:55.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:55.546+0000 7f488f0e6700 1 -- 172.21.15.146:0/2548990974 shutdown_connections 2024-01-21T20:07:55.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:55.546+0000 7f488f0e6700 1 -- 172.21.15.146:0/2548990974 wait complete. 2024-01-21T20:07:55.537 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:07:56.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:56 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:07:55.209980+0000 mgr.smithi053.fgsmol (mgr.14164) 271 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:56.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:56 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:07:55.545396+0000 mon.smithi053 (mon.0) 316 : audit [DBG] from='client.? 172.21.15.146:0/2548990974' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:07:56.949 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:07:56.949 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:07:58.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:58.181+0000 7f4cc359e700 1 -- 172.21.15.146:0/124014608 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cc40f62a0 msgr2=0x7f4cc40f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:58.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:58.181+0000 7f4cc359e700 1 --2- 172.21.15.146:0/124014608 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cc40f62a0 0x7f4cc40f66c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f4cac009fb0 tx=0x7f4cac009b30).stop 2024-01-21T20:07:58.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:58.181+0000 7f4cc359e700 1 -- 172.21.15.146:0/124014608 shutdown_connections 2024-01-21T20:07:58.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:58.181+0000 7f4cc359e700 1 --2- 172.21.15.146:0/124014608 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cc40f62a0 0x7f4cc40f66c0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:58.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:58.181+0000 7f4cc359e700 1 -- 172.21.15.146:0/124014608 >> 172.21.15.146:0/124014608 conn(0x7f4cc40f1aa0 msgr2=0x7f4cc40f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:58.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:58.182+0000 7f4cc359e700 1 -- 172.21.15.146:0/124014608 shutdown_connections 2024-01-21T20:07:58.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:58.182+0000 7f4cc359e700 1 -- 172.21.15.146:0/124014608 wait complete. 2024-01-21T20:07:58.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:58.182+0000 7f4cc359e700 1 Processor -- start 2024-01-21T20:07:58.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:58.182+0000 7f4cc359e700 1 -- start start 2024-01-21T20:07:58.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:58.182+0000 7f4cc359e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cc40f62a0 0x7f4cc41064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:58.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:58.182+0000 7f4cc359e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc41069a0 con 0x7f4cc40f62a0 2024-01-21T20:07:58.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:58.183+0000 7f4cc259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cc40f62a0 0x7f4cc41064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:58.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:58.183+0000 7f4cc259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cc40f62a0 0x7f4cc41064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:49402/0 (socket says 172.21.15.146:49402) 2024-01-21T20:07:58.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:58.183+0000 7f4cc259c700 1 -- 172.21.15.146:0/3812300444 learned_addr learned my addr 172.21.15.146:0/3812300444 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:07:58.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:58.183+0000 7f4cc259c700 1 -- 172.21.15.146:0/3812300444 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cac005040 con 0x7f4cc40f62a0 2024-01-21T20:07:58.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:58.183+0000 7f4cc259c700 1 --2- 172.21.15.146:0/3812300444 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cc40f62a0 0x7f4cc41064b0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f4cac013510 tx=0x7f4cac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:58.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:58.183+0000 7f4cbaffd700 1 -- 172.21.15.146:0/3812300444 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4cac009320 con 0x7f4cc40f62a0 2024-01-21T20:07:58.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:58.183+0000 7f4cc359e700 1 -- 172.21.15.146:0/3812300444 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cc4106ba0 con 0x7f4cc40f62a0 2024-01-21T20:07:58.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:58.183+0000 7f4cbaffd700 1 -- 172.21.15.146:0/3812300444 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4cac013f00 con 0x7f4cc40f62a0 2024-01-21T20:07:58.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:58.183+0000 7f4cbaffd700 1 -- 172.21.15.146:0/3812300444 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4cac014dc0 con 0x7f4cc40f62a0 2024-01-21T20:07:58.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:58.183+0000 7f4cc359e700 1 -- 172.21.15.146:0/3812300444 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cc4107030 con 0x7f4cc40f62a0 2024-01-21T20:07:58.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:58.184+0000 7f4cbaffd700 1 -- 172.21.15.146:0/3812300444 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4cac0095f0 con 0x7f4cc40f62a0 2024-01-21T20:07:58.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:58.184+0000 7f4cc359e700 1 -- 172.21.15.146:0/3812300444 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cc4045ad0 con 0x7f4cc40f62a0 2024-01-21T20:07:58.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:58.184+0000 7f4cbaffd700 1 --2- 172.21.15.146:0/3812300444 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f4cb002ff60 0x7f4cb0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:58.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:58.184+0000 7f4cbaffd700 1 -- 172.21.15.146:0/3812300444 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4cac041000 con 0x7f4cc40f62a0 2024-01-21T20:07:58.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:58.184+0000 7f4cc1d9b700 1 --2- 172.21.15.146:0/3812300444 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f4cb002ff60 0x7f4cb0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:58.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:58.185+0000 7f4cc1d9b700 1 --2- 172.21.15.146:0/3812300444 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f4cb002ff60 0x7f4cb0032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f4cb40096a0 tx=0x7f4cb4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:58.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:58.189+0000 7f4cbaffd700 1 -- 172.21.15.146:0/3812300444 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4cac01c790 con 0x7f4cc40f62a0 2024-01-21T20:07:58.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:58.393+0000 7f4cc359e700 1 -- 172.21.15.146:0/3812300444 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4cc4107ee0 con 0x7f4cc40f62a0 2024-01-21T20:07:58.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:58.394+0000 7f4cbaffd700 1 -- 172.21.15.146:0/3812300444 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4cac010230 con 0x7f4cc40f62a0 2024-01-21T20:07:58.384 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:07:58.385 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:07:58.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:58.395+0000 7f4cc359e700 1 -- 172.21.15.146:0/3812300444 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f4cb002ff60 msgr2=0x7f4cb0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:58.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:58.395+0000 7f4cc359e700 1 --2- 172.21.15.146:0/3812300444 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f4cb002ff60 0x7f4cb0032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f4cb40096a0 tx=0x7f4cb4006b40).stop 2024-01-21T20:07:58.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:58.395+0000 7f4cc359e700 1 -- 172.21.15.146:0/3812300444 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cc40f62a0 msgr2=0x7f4cc41064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:58.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:58.395+0000 7f4cc359e700 1 --2- 172.21.15.146:0/3812300444 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cc40f62a0 0x7f4cc41064b0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f4cac013510 tx=0x7f4cac00b040).stop 2024-01-21T20:07:58.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:58.395+0000 7f4cc359e700 1 -- 172.21.15.146:0/3812300444 shutdown_connections 2024-01-21T20:07:58.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:58.395+0000 7f4cc359e700 1 --2- 172.21.15.146:0/3812300444 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f4cb002ff60 0x7f4cb0032420 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:58.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:58.395+0000 7f4cc359e700 1 --2- 172.21.15.146:0/3812300444 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cc40f62a0 0x7f4cc41064b0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:58.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:58.395+0000 7f4cc359e700 1 -- 172.21.15.146:0/3812300444 >> 172.21.15.146:0/3812300444 conn(0x7f4cc40f1aa0 msgr2=0x7f4cc40f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:58.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:58.395+0000 7f4cc359e700 1 -- 172.21.15.146:0/3812300444 shutdown_connections 2024-01-21T20:07:58.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:07:58.395+0000 7f4cc359e700 1 -- 172.21.15.146:0/3812300444 wait complete. 2024-01-21T20:07:58.387 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:07:58.932 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:58 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:07:57.210283+0000 mgr.smithi053.fgsmol (mgr.14164) 272 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:58.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:07:58 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:07:58.395301+0000 mon.smithi053 (mon.0) 317 : audit [DBG] from='client.? 172.21.15.146:0/3812300444' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:07:59.821 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:07:59.821 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:08:00.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:08:00 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:07:59.210591+0000 mgr.smithi053.fgsmol (mgr.14164) 273 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:08:01.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:01.096+0000 7f0344d44700 1 -- 172.21.15.146:0/1991216753 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f03400f6510 msgr2=0x7f03400f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:01.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:01.096+0000 7f0344d44700 1 --2- 172.21.15.146:0/1991216753 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f03400f6510 0x7f03400f6930 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7f0328009fb0 tx=0x7f0328009b30).stop 2024-01-21T20:08:01.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:01.096+0000 7f0344d44700 1 -- 172.21.15.146:0/1991216753 shutdown_connections 2024-01-21T20:08:01.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:01.096+0000 7f0344d44700 1 --2- 172.21.15.146:0/1991216753 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f03400f6510 0x7f03400f6930 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:01.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:01.096+0000 7f0344d44700 1 -- 172.21.15.146:0/1991216753 >> 172.21.15.146:0/1991216753 conn(0x7f03400f1c70 msgr2=0x7f03400f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:01.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:01.096+0000 7f0344d44700 1 -- 172.21.15.146:0/1991216753 shutdown_connections 2024-01-21T20:08:01.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:01.096+0000 7f0344d44700 1 -- 172.21.15.146:0/1991216753 wait complete. 2024-01-21T20:08:01.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:01.097+0000 7f0344d44700 1 Processor -- start 2024-01-21T20:08:01.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:01.097+0000 7f0344d44700 1 -- start start 2024-01-21T20:08:01.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:01.097+0000 7f0344d44700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f03400f6510 0x7f0340106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:01.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:01.097+0000 7f0344d44700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0340106c20 con 0x7f03400f6510 2024-01-21T20:08:01.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:01.097+0000 7f033e59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f03400f6510 0x7f0340106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:01.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:01.097+0000 7f033e59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f03400f6510 0x7f0340106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:49416/0 (socket says 172.21.15.146:49416) 2024-01-21T20:08:01.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:01.097+0000 7f033e59c700 1 -- 172.21.15.146:0/3505205250 learned_addr learned my addr 172.21.15.146:0/3505205250 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:08:01.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:01.098+0000 7f033e59c700 1 -- 172.21.15.146:0/3505205250 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0328005040 con 0x7f03400f6510 2024-01-21T20:08:01.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:01.098+0000 7f033e59c700 1 --2- 172.21.15.146:0/3505205250 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f03400f6510 0x7f0340106730 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f0328013510 tx=0x7f032800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:01.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:01.098+0000 7f0336ffd700 1 -- 172.21.15.146:0/3505205250 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0328009320 con 0x7f03400f6510 2024-01-21T20:08:01.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:01.098+0000 7f0344d44700 1 -- 172.21.15.146:0/3505205250 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0340106e20 con 0x7f03400f6510 2024-01-21T20:08:01.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:01.098+0000 7f0336ffd700 1 -- 172.21.15.146:0/3505205250 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0328013f00 con 0x7f03400f6510 2024-01-21T20:08:01.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:01.098+0000 7f0336ffd700 1 -- 172.21.15.146:0/3505205250 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0328014d90 con 0x7f03400f6510 2024-01-21T20:08:01.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:01.098+0000 7f0344d44700 1 -- 172.21.15.146:0/3505205250 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03401072b0 con 0x7f03400f6510 2024-01-21T20:08:01.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:01.099+0000 7f0344d44700 1 -- 172.21.15.146:0/3505205250 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0340045ad0 con 0x7f03400f6510 2024-01-21T20:08:01.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:01.099+0000 7f0336ffd700 1 -- 172.21.15.146:0/3505205250 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f03280095f0 con 0x7f03400f6510 2024-01-21T20:08:01.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:01.099+0000 7f0336ffd700 1 --2- 172.21.15.146:0/3505205250 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f032c02ff60 0x7f032c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:01.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:01.099+0000 7f0336ffd700 1 -- 172.21.15.146:0/3505205250 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0328041000 con 0x7f03400f6510 2024-01-21T20:08:01.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:01.100+0000 7f033dd9b700 1 --2- 172.21.15.146:0/3505205250 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f032c02ff60 0x7f032c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:01.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:01.100+0000 7f033dd9b700 1 --2- 172.21.15.146:0/3505205250 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f032c02ff60 0x7f032c032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f03300096a0 tx=0x7f0330006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:01.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:01.104+0000 7f0336ffd700 1 -- 172.21.15.146:0/3505205250 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0328014420 con 0x7f03400f6510 2024-01-21T20:08:01.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:01.311+0000 7f0344d44700 1 -- 172.21.15.146:0/3505205250 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f03400fbf60 con 0x7f03400f6510 2024-01-21T20:08:01.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:01.312+0000 7f0336ffd700 1 -- 172.21.15.146:0/3505205250 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0328010230 con 0x7f03400f6510 2024-01-21T20:08:01.302 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:08:01.303 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:08:01.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:01.313+0000 7f0344d44700 1 -- 172.21.15.146:0/3505205250 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f032c02ff60 msgr2=0x7f032c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:01.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:01.313+0000 7f0344d44700 1 --2- 172.21.15.146:0/3505205250 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f032c02ff60 0x7f032c032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f03300096a0 tx=0x7f0330006b40).stop 2024-01-21T20:08:01.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:01.313+0000 7f0344d44700 1 -- 172.21.15.146:0/3505205250 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f03400f6510 msgr2=0x7f0340106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:01.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:01.313+0000 7f0344d44700 1 --2- 172.21.15.146:0/3505205250 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f03400f6510 0x7f0340106730 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f0328013510 tx=0x7f032800b040).stop 2024-01-21T20:08:01.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:01.313+0000 7f0344d44700 1 -- 172.21.15.146:0/3505205250 shutdown_connections 2024-01-21T20:08:01.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:01.313+0000 7f0344d44700 1 --2- 172.21.15.146:0/3505205250 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f032c02ff60 0x7f032c032420 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:01.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:01.313+0000 7f0344d44700 1 --2- 172.21.15.146:0/3505205250 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f03400f6510 0x7f0340106730 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:01.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:01.313+0000 7f0344d44700 1 -- 172.21.15.146:0/3505205250 >> 172.21.15.146:0/3505205250 conn(0x7f03400f1c70 msgr2=0x7f03400f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:01.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:01.314+0000 7f0344d44700 1 -- 172.21.15.146:0/3505205250 shutdown_connections 2024-01-21T20:08:01.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:01.314+0000 7f0344d44700 1 -- 172.21.15.146:0/3505205250 wait complete. 2024-01-21T20:08:01.305 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:08:01.932 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:08:01 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:08:01.313281+0000 mon.smithi053 (mon.0) 318 : audit [DBG] from='client.? 172.21.15.146:0/3505205250' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:08:02.702 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:08:02.702 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:08:02.932 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:08:02 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:08:01.210942+0000 mgr.smithi053.fgsmol (mgr.14164) 274 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:08:03.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:03.894+0000 7f04e0138700 1 -- 172.21.15.146:0/95674921 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04d80f53f0 msgr2=0x7f04d80f5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:03.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:03.894+0000 7f04e0138700 1 --2- 172.21.15.146:0/95674921 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04d80f53f0 0x7f04d80f5810 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7f04c8009fb0 tx=0x7f04c8009b30).stop 2024-01-21T20:08:03.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:03.894+0000 7f04e0138700 1 -- 172.21.15.146:0/95674921 shutdown_connections 2024-01-21T20:08:03.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:03.894+0000 7f04e0138700 1 --2- 172.21.15.146:0/95674921 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04d80f53f0 0x7f04d80f5810 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:03.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:03.894+0000 7f04e0138700 1 -- 172.21.15.146:0/95674921 >> 172.21.15.146:0/95674921 conn(0x7f04d80f0b70 msgr2=0x7f04d80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:03.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:03.894+0000 7f04e0138700 1 -- 172.21.15.146:0/95674921 shutdown_connections 2024-01-21T20:08:03.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:03.894+0000 7f04e0138700 1 -- 172.21.15.146:0/95674921 wait complete. 2024-01-21T20:08:03.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:03.895+0000 7f04e0138700 1 Processor -- start 2024-01-21T20:08:03.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:03.895+0000 7f04e0138700 1 -- start start 2024-01-21T20:08:03.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:03.895+0000 7f04e0138700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04d80f53f0 0x7f04d81055e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:03.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:03.895+0000 7f04e0138700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04d8105ad0 con 0x7f04d80f53f0 2024-01-21T20:08:03.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:03.895+0000 7f04dded4700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04d80f53f0 0x7f04d81055e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:03.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:03.895+0000 7f04dded4700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04d80f53f0 0x7f04d81055e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:49448/0 (socket says 172.21.15.146:49448) 2024-01-21T20:08:03.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:03.895+0000 7f04dded4700 1 -- 172.21.15.146:0/2044696776 learned_addr learned my addr 172.21.15.146:0/2044696776 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:08:03.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:03.896+0000 7f04dded4700 1 -- 172.21.15.146:0/2044696776 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04c8005040 con 0x7f04d80f53f0 2024-01-21T20:08:03.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:03.896+0000 7f04dded4700 1 --2- 172.21.15.146:0/2044696776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04d80f53f0 0x7f04d81055e0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f04c8013510 tx=0x7f04c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:03.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:03.896+0000 7f04ce7fc700 1 -- 172.21.15.146:0/2044696776 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f04c8009320 con 0x7f04d80f53f0 2024-01-21T20:08:03.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:03.896+0000 7f04e0138700 1 -- 172.21.15.146:0/2044696776 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04d8105cd0 con 0x7f04d80f53f0 2024-01-21T20:08:03.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:03.896+0000 7f04ce7fc700 1 -- 172.21.15.146:0/2044696776 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f04c8013f00 con 0x7f04d80f53f0 2024-01-21T20:08:03.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:03.896+0000 7f04ce7fc700 1 -- 172.21.15.146:0/2044696776 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f04c8014d90 con 0x7f04d80f53f0 2024-01-21T20:08:03.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:03.896+0000 7f04e0138700 1 -- 172.21.15.146:0/2044696776 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04d8106160 con 0x7f04d80f53f0 2024-01-21T20:08:03.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:03.897+0000 7f04ce7fc700 1 -- 172.21.15.146:0/2044696776 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f04c80095f0 con 0x7f04d80f53f0 2024-01-21T20:08:03.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:03.897+0000 7f04e0138700 1 -- 172.21.15.146:0/2044696776 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04d8045ad0 con 0x7f04d80f53f0 2024-01-21T20:08:03.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:03.897+0000 7f04ce7fc700 1 --2- 172.21.15.146:0/2044696776 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f04c402ff60 0x7f04c4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:03.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:03.897+0000 7f04ce7fc700 1 -- 172.21.15.146:0/2044696776 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f04c8040fe0 con 0x7f04d80f53f0 2024-01-21T20:08:03.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:03.897+0000 7f04dd6d3700 1 --2- 172.21.15.146:0/2044696776 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f04c402ff60 0x7f04c4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:03.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:03.897+0000 7f04dd6d3700 1 --2- 172.21.15.146:0/2044696776 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f04c402ff60 0x7f04c4032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f04d40096a0 tx=0x7f04d4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:03.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:03.902+0000 7f04ce7fc700 1 -- 172.21.15.146:0/2044696776 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f04c801d7f0 con 0x7f04d80f53f0 2024-01-21T20:08:04.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:04.107+0000 7f04e0138700 1 -- 172.21.15.146:0/2044696776 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f04d8107010 con 0x7f04d80f53f0 2024-01-21T20:08:04.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:04.108+0000 7f04ce7fc700 1 -- 172.21.15.146:0/2044696776 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f04c8010230 con 0x7f04d80f53f0 2024-01-21T20:08:04.098 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:08:04.098 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:08:04.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:04.109+0000 7f04e0138700 1 -- 172.21.15.146:0/2044696776 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f04c402ff60 msgr2=0x7f04c4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:04.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:04.109+0000 7f04e0138700 1 --2- 172.21.15.146:0/2044696776 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f04c402ff60 0x7f04c4032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f04d40096a0 tx=0x7f04d4006b40).stop 2024-01-21T20:08:04.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:04.109+0000 7f04e0138700 1 -- 172.21.15.146:0/2044696776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04d80f53f0 msgr2=0x7f04d81055e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:04.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:04.109+0000 7f04e0138700 1 --2- 172.21.15.146:0/2044696776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04d80f53f0 0x7f04d81055e0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f04c8013510 tx=0x7f04c800b040).stop 2024-01-21T20:08:04.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:04.109+0000 7f04e0138700 1 -- 172.21.15.146:0/2044696776 shutdown_connections 2024-01-21T20:08:04.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:04.109+0000 7f04e0138700 1 --2- 172.21.15.146:0/2044696776 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f04c402ff60 0x7f04c4032420 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:04.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:04.109+0000 7f04e0138700 1 --2- 172.21.15.146:0/2044696776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04d80f53f0 0x7f04d81055e0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:04.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:04.109+0000 7f04e0138700 1 -- 172.21.15.146:0/2044696776 >> 172.21.15.146:0/2044696776 conn(0x7f04d80f0b70 msgr2=0x7f04d80f1640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:04.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:04.109+0000 7f04e0138700 1 -- 172.21.15.146:0/2044696776 shutdown_connections 2024-01-21T20:08:04.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:04.109+0000 7f04e0138700 1 -- 172.21.15.146:0/2044696776 wait complete. 2024-01-21T20:08:04.101 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:08:04.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:08:04 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:08:03.211351+0000 mgr.smithi053.fgsmol (mgr.14164) 275 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:08:04.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:08:04 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:08:04.109104+0000 mon.smithi053 (mon.0) 319 : audit [DBG] from='client.? 172.21.15.146:0/2044696776' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:08:05.526 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:08:05.527 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:08:06.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:06.740+0000 7fcadd5ad700 1 -- 172.21.15.146:0/834003767 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcad80f7b60 msgr2=0x7fcad80f7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:06.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:06.740+0000 7fcadd5ad700 1 --2- 172.21.15.146:0/834003767 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcad80f7b60 0x7fcad80f7f80 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7fcac8009fb0 tx=0x7fcac8009b30).stop 2024-01-21T20:08:06.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:06.740+0000 7fcadd5ad700 1 -- 172.21.15.146:0/834003767 shutdown_connections 2024-01-21T20:08:06.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:06.740+0000 7fcadd5ad700 1 --2- 172.21.15.146:0/834003767 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcad80f7b60 0x7fcad80f7f80 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:06.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:06.740+0000 7fcadd5ad700 1 -- 172.21.15.146:0/834003767 >> 172.21.15.146:0/834003767 conn(0x7fcad80f02e0 msgr2=0x7fcad80f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:06.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:06.740+0000 7fcadd5ad700 1 -- 172.21.15.146:0/834003767 shutdown_connections 2024-01-21T20:08:06.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:06.740+0000 7fcadd5ad700 1 -- 172.21.15.146:0/834003767 wait complete. 2024-01-21T20:08:06.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:06.741+0000 7fcadd5ad700 1 Processor -- start 2024-01-21T20:08:06.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:06.741+0000 7fcadd5ad700 1 -- start start 2024-01-21T20:08:06.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:06.741+0000 7fcadd5ad700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcad80f7b60 0x7fcad8106740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:06.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:06.741+0000 7fcadd5ad700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad8106c30 con 0x7fcad80f7b60 2024-01-21T20:08:06.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:06.742+0000 7fcad6ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcad80f7b60 0x7fcad8106740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:06.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:06.742+0000 7fcad6ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcad80f7b60 0x7fcad8106740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:49466/0 (socket says 172.21.15.146:49466) 2024-01-21T20:08:06.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:06.742+0000 7fcad6ffd700 1 -- 172.21.15.146:0/3312009243 learned_addr learned my addr 172.21.15.146:0/3312009243 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:08:06.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:06.742+0000 7fcad6ffd700 1 -- 172.21.15.146:0/3312009243 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac8005040 con 0x7fcad80f7b60 2024-01-21T20:08:06.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:06.742+0000 7fcad6ffd700 1 --2- 172.21.15.146:0/3312009243 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcad80f7b60 0x7fcad8106740 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7fcac8013510 tx=0x7fcac800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:06.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:06.742+0000 7fcabf7fe700 1 -- 172.21.15.146:0/3312009243 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcac8009320 con 0x7fcad80f7b60 2024-01-21T20:08:06.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:06.742+0000 7fcabf7fe700 1 -- 172.21.15.146:0/3312009243 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcac8013f00 con 0x7fcad80f7b60 2024-01-21T20:08:06.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:06.742+0000 7fcadd5ad700 1 -- 172.21.15.146:0/3312009243 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcad8106e30 con 0x7fcad80f7b60 2024-01-21T20:08:06.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:06.742+0000 7fcabf7fe700 1 -- 172.21.15.146:0/3312009243 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcac8014d90 con 0x7fcad80f7b60 2024-01-21T20:08:06.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:06.742+0000 7fcadd5ad700 1 -- 172.21.15.146:0/3312009243 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcad81072c0 con 0x7fcad80f7b60 2024-01-21T20:08:06.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:06.743+0000 7fcabf7fe700 1 -- 172.21.15.146:0/3312009243 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcac80263a0 con 0x7fcad80f7b60 2024-01-21T20:08:06.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:06.743+0000 7fcabf7fe700 1 --2- 172.21.15.146:0/3312009243 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fcac002ff60 0x7fcac0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:06.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:06.743+0000 7fcadd5ad700 1 -- 172.21.15.146:0/3312009243 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcad8045ad0 con 0x7fcad80f7b60 2024-01-21T20:08:06.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:06.743+0000 7fcabf7fe700 1 -- 172.21.15.146:0/3312009243 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcac8041380 con 0x7fcad80f7b60 2024-01-21T20:08:06.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:06.744+0000 7fcad67fc700 1 --2- 172.21.15.146:0/3312009243 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fcac002ff60 0x7fcac0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:06.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:06.744+0000 7fcad67fc700 1 --2- 172.21.15.146:0/3312009243 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fcac002ff60 0x7fcac0032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fcacc0096a0 tx=0x7fcacc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:06.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:06.749+0000 7fcabf7fe700 1 -- 172.21.15.146:0/3312009243 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcac8012920 con 0x7fcad80f7b60 2024-01-21T20:08:06.932 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:08:06 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:08:05.211658+0000 mgr.smithi053.fgsmol (mgr.14164) 276 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:08:06.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:06.953+0000 7fcadd5ad700 1 -- 172.21.15.146:0/3312009243 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcad8108170 con 0x7fcad80f7b60 2024-01-21T20:08:06.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:06.953+0000 7fcabf7fe700 1 -- 172.21.15.146:0/3312009243 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcac8012330 con 0x7fcad80f7b60 2024-01-21T20:08:06.944 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:08:06.944 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:08:06.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:06.955+0000 7fcadd5ad700 1 -- 172.21.15.146:0/3312009243 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fcac002ff60 msgr2=0x7fcac0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:06.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:06.955+0000 7fcadd5ad700 1 --2- 172.21.15.146:0/3312009243 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fcac002ff60 0x7fcac0032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fcacc0096a0 tx=0x7fcacc006b40).stop 2024-01-21T20:08:06.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:06.955+0000 7fcadd5ad700 1 -- 172.21.15.146:0/3312009243 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcad80f7b60 msgr2=0x7fcad8106740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:06.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:06.955+0000 7fcadd5ad700 1 --2- 172.21.15.146:0/3312009243 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcad80f7b60 0x7fcad8106740 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7fcac8013510 tx=0x7fcac800b040).stop 2024-01-21T20:08:06.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:06.955+0000 7fcadd5ad700 1 -- 172.21.15.146:0/3312009243 shutdown_connections 2024-01-21T20:08:06.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:06.955+0000 7fcadd5ad700 1 --2- 172.21.15.146:0/3312009243 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fcac002ff60 0x7fcac0032420 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:06.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:06.955+0000 7fcadd5ad700 1 --2- 172.21.15.146:0/3312009243 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcad80f7b60 0x7fcad8106740 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:06.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:06.955+0000 7fcadd5ad700 1 -- 172.21.15.146:0/3312009243 >> 172.21.15.146:0/3312009243 conn(0x7fcad80f02e0 msgr2=0x7fcad80f0e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:06.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:06.955+0000 7fcadd5ad700 1 -- 172.21.15.146:0/3312009243 shutdown_connections 2024-01-21T20:08:06.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:06.955+0000 7fcadd5ad700 1 -- 172.21.15.146:0/3312009243 wait complete. 2024-01-21T20:08:06.947 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:08:07.932 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:08:07 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:08:06.954891+0000 mon.smithi053 (mon.0) 320 : audit [DBG] from='client.? 172.21.15.146:0/3312009243' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:08:08.335 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:08:08.336 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:08:08.932 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:08:08 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:08:07.212077+0000 mgr.smithi053.fgsmol (mgr.14164) 277 : cluster [DBG] pgmap v263: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:08:09.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:09.530+0000 7fc187fff700 1 -- 172.21.15.146:0/2835704707 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1880601e0 msgr2=0x7fc1880f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:09.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:09.530+0000 7fc187fff700 1 --2- 172.21.15.146:0/2835704707 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1880601e0 0x7fc1880f8f20 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7fc17c009fb0 tx=0x7fc17c009b30).stop 2024-01-21T20:08:09.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:09.530+0000 7fc187fff700 1 -- 172.21.15.146:0/2835704707 shutdown_connections 2024-01-21T20:08:09.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:09.530+0000 7fc187fff700 1 --2- 172.21.15.146:0/2835704707 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1880601e0 0x7fc1880f8f20 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:09.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:09.530+0000 7fc187fff700 1 -- 172.21.15.146:0/2835704707 >> 172.21.15.146:0/2835704707 conn(0x7fc1880f09a0 msgr2=0x7fc1880f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:09.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:09.530+0000 7fc187fff700 1 -- 172.21.15.146:0/2835704707 shutdown_connections 2024-01-21T20:08:09.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:09.530+0000 7fc187fff700 1 -- 172.21.15.146:0/2835704707 wait complete. 2024-01-21T20:08:09.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:09.530+0000 7fc187fff700 1 Processor -- start 2024-01-21T20:08:09.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:09.530+0000 7fc187fff700 1 -- start start 2024-01-21T20:08:09.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:09.531+0000 7fc187fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1880601e0 0x7fc1880f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:09.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:09.531+0000 7fc187fff700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1880f4e20 con 0x7fc1880601e0 2024-01-21T20:08:09.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:09.531+0000 7fc186ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1880601e0 0x7fc1880f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:09.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:09.531+0000 7fc186ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1880601e0 0x7fc1880f69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:49740/0 (socket says 172.21.15.146:49740) 2024-01-21T20:08:09.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:09.531+0000 7fc186ffd700 1 -- 172.21.15.146:0/1011569501 learned_addr learned my addr 172.21.15.146:0/1011569501 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:08:09.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:09.531+0000 7fc186ffd700 1 -- 172.21.15.146:0/1011569501 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc17c005040 con 0x7fc1880601e0 2024-01-21T20:08:09.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:09.531+0000 7fc186ffd700 1 --2- 172.21.15.146:0/1011569501 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1880601e0 0x7fc1880f69a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fc17c013510 tx=0x7fc17c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:09.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:09.531+0000 7fc16f7fe700 1 -- 172.21.15.146:0/1011569501 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc17c009320 con 0x7fc1880601e0 2024-01-21T20:08:09.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:09.532+0000 7fc187fff700 1 -- 172.21.15.146:0/1011569501 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1880f4fc0 con 0x7fc1880601e0 2024-01-21T20:08:09.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:09.532+0000 7fc16f7fe700 1 -- 172.21.15.146:0/1011569501 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc17c013f00 con 0x7fc1880601e0 2024-01-21T20:08:09.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:09.532+0000 7fc16f7fe700 1 -- 172.21.15.146:0/1011569501 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc17c014d90 con 0x7fc1880601e0 2024-01-21T20:08:09.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:09.532+0000 7fc187fff700 1 -- 172.21.15.146:0/1011569501 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1880f5450 con 0x7fc1880601e0 2024-01-21T20:08:09.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:09.532+0000 7fc16f7fe700 1 -- 172.21.15.146:0/1011569501 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc17c0095f0 con 0x7fc1880601e0 2024-01-21T20:08:09.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:09.532+0000 7fc187fff700 1 -- 172.21.15.146:0/1011569501 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc188059100 con 0x7fc1880601e0 2024-01-21T20:08:09.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:09.533+0000 7fc16f7fe700 1 --2- 172.21.15.146:0/1011569501 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fc17002ff60 0x7fc170032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:09.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:09.533+0000 7fc16f7fe700 1 -- 172.21.15.146:0/1011569501 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc17c041000 con 0x7fc1880601e0 2024-01-21T20:08:09.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:09.533+0000 7fc1867fc700 1 --2- 172.21.15.146:0/1011569501 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fc17002ff60 0x7fc170032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:09.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:09.533+0000 7fc1867fc700 1 --2- 172.21.15.146:0/1011569501 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fc17002ff60 0x7fc170032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fc1780096a0 tx=0x7fc178006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:09.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:09.537+0000 7fc16f7fe700 1 -- 172.21.15.146:0/1011569501 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc17c014420 con 0x7fc1880601e0 2024-01-21T20:08:09.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:09.746+0000 7fc187fff700 1 -- 172.21.15.146:0/1011569501 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc1880fbcd0 con 0x7fc1880601e0 2024-01-21T20:08:09.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:09.747+0000 7fc16f7fe700 1 -- 172.21.15.146:0/1011569501 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc17c010230 con 0x7fc1880601e0 2024-01-21T20:08:09.738 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:08:09.738 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:08:09.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:09.748+0000 7fc187fff700 1 -- 172.21.15.146:0/1011569501 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fc17002ff60 msgr2=0x7fc170032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:09.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:09.748+0000 7fc187fff700 1 --2- 172.21.15.146:0/1011569501 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fc17002ff60 0x7fc170032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fc1780096a0 tx=0x7fc178006b40).stop 2024-01-21T20:08:09.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:09.748+0000 7fc187fff700 1 -- 172.21.15.146:0/1011569501 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1880601e0 msgr2=0x7fc1880f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:09.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:09.748+0000 7fc187fff700 1 --2- 172.21.15.146:0/1011569501 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1880601e0 0x7fc1880f69a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fc17c013510 tx=0x7fc17c00b040).stop 2024-01-21T20:08:09.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:09.748+0000 7fc187fff700 1 -- 172.21.15.146:0/1011569501 shutdown_connections 2024-01-21T20:08:09.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:09.748+0000 7fc187fff700 1 --2- 172.21.15.146:0/1011569501 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fc17002ff60 0x7fc170032420 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:09.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:09.748+0000 7fc187fff700 1 --2- 172.21.15.146:0/1011569501 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1880601e0 0x7fc1880f69a0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:09.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:09.748+0000 7fc187fff700 1 -- 172.21.15.146:0/1011569501 >> 172.21.15.146:0/1011569501 conn(0x7fc1880f09a0 msgr2=0x7fc1880f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:09.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:09.748+0000 7fc187fff700 1 -- 172.21.15.146:0/1011569501 shutdown_connections 2024-01-21T20:08:09.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:09.748+0000 7fc187fff700 1 -- 172.21.15.146:0/1011569501 wait complete. 2024-01-21T20:08:09.740 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:08:09.932 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:08:09 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:08:09.273244+0000 mon.smithi053 (mon.0) 321 : audit [INF] from='mgr.14164 172.21.15.53:0/2331671680' entity='mgr.smithi053.fgsmol' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fgsmol/mirror_snapshot_schedule"}]: dispatch 2024-01-21T20:08:09.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:08:09 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:08:09.279997+0000 mon.smithi053 (mon.0) 322 : audit [INF] from='mgr.14164 172.21.15.53:0/2331671680' entity='mgr.smithi053.fgsmol' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fgsmol/trash_purge_schedule"}]: dispatch 2024-01-21T20:08:10.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:08:10 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:08:09.212452+0000 mgr.smithi053.fgsmol (mgr.14164) 278 : cluster [DBG] pgmap v264: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:08:10.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:08:10 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:08:09.748193+0000 mon.smithi053 (mon.0) 323 : audit [DBG] from='client.? 172.21.15.146:0/1011569501' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:08:11.151 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:08:11.151 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:08:12.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:12.399+0000 7f7b64377700 1 -- 172.21.15.146:0/2402836475 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b5c0f47c0 msgr2=0x7f7b5c0f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:12.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:12.399+0000 7f7b64377700 1 --2- 172.21.15.146:0/2402836475 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b5c0f47c0 0x7f7b5c0f4be0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f7b58009fb0 tx=0x7f7b58009b30).stop 2024-01-21T20:08:12.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:12.399+0000 7f7b64377700 1 -- 172.21.15.146:0/2402836475 shutdown_connections 2024-01-21T20:08:12.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:12.399+0000 7f7b64377700 1 --2- 172.21.15.146:0/2402836475 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b5c0f47c0 0x7f7b5c0f4be0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:12.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:12.399+0000 7f7b64377700 1 -- 172.21.15.146:0/2402836475 >> 172.21.15.146:0/2402836475 conn(0x7f7b5c0f02e0 msgr2=0x7f7b5c0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:12.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:12.399+0000 7f7b64377700 1 -- 172.21.15.146:0/2402836475 shutdown_connections 2024-01-21T20:08:12.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:12.399+0000 7f7b64377700 1 -- 172.21.15.146:0/2402836475 wait complete. 2024-01-21T20:08:12.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:12.400+0000 7f7b64377700 1 Processor -- start 2024-01-21T20:08:12.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:12.400+0000 7f7b64377700 1 -- start start 2024-01-21T20:08:12.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:12.400+0000 7f7b64377700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b5c0f47c0 0x7f7b5c106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:12.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:12.400+0000 7f7b64377700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b5c106bf0 con 0x7f7b5c0f47c0 2024-01-21T20:08:12.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:12.400+0000 7f7b62113700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b5c0f47c0 0x7f7b5c106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:12.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:12.400+0000 7f7b62113700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b5c0f47c0 0x7f7b5c106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:49762/0 (socket says 172.21.15.146:49762) 2024-01-21T20:08:12.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:12.400+0000 7f7b62113700 1 -- 172.21.15.146:0/731692716 learned_addr learned my addr 172.21.15.146:0/731692716 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:08:12.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:12.400+0000 7f7b62113700 1 -- 172.21.15.146:0/731692716 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b58005040 con 0x7f7b5c0f47c0 2024-01-21T20:08:12.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:12.400+0000 7f7b62113700 1 --2- 172.21.15.146:0/731692716 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b5c0f47c0 0x7f7b5c106700 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f7b58013510 tx=0x7f7b5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:12.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:12.401+0000 7f7b52ffd700 1 -- 172.21.15.146:0/731692716 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b58009320 con 0x7f7b5c0f47c0 2024-01-21T20:08:12.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:12.401+0000 7f7b52ffd700 1 -- 172.21.15.146:0/731692716 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7b58013f00 con 0x7f7b5c0f47c0 2024-01-21T20:08:12.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:12.401+0000 7f7b64377700 1 -- 172.21.15.146:0/731692716 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b5c106df0 con 0x7f7b5c0f47c0 2024-01-21T20:08:12.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:12.401+0000 7f7b52ffd700 1 -- 172.21.15.146:0/731692716 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b58014d90 con 0x7f7b5c0f47c0 2024-01-21T20:08:12.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:12.401+0000 7f7b64377700 1 -- 172.21.15.146:0/731692716 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b5c107280 con 0x7f7b5c0f47c0 2024-01-21T20:08:12.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:12.401+0000 7f7b52ffd700 1 -- 172.21.15.146:0/731692716 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7b58014420 con 0x7f7b5c0f47c0 2024-01-21T20:08:12.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:12.401+0000 7f7b64377700 1 -- 172.21.15.146:0/731692716 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b5c045ad0 con 0x7f7b5c0f47c0 2024-01-21T20:08:12.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:12.402+0000 7f7b52ffd700 1 --2- 172.21.15.146:0/731692716 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f7b4802ff60 0x7f7b48032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:12.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:12.402+0000 7f7b52ffd700 1 -- 172.21.15.146:0/731692716 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7b58040fb0 con 0x7f7b5c0f47c0 2024-01-21T20:08:12.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:12.402+0000 7f7b61912700 1 --2- 172.21.15.146:0/731692716 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f7b4802ff60 0x7f7b48032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:12.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:12.402+0000 7f7b61912700 1 --2- 172.21.15.146:0/731692716 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f7b4802ff60 0x7f7b48032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f7b4c0096a0 tx=0x7f7b4c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:12.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:12.406+0000 7f7b52ffd700 1 -- 172.21.15.146:0/731692716 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7b580095f0 con 0x7f7b5c0f47c0 2024-01-21T20:08:12.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:12.609+0000 7f7b64377700 1 -- 172.21.15.146:0/731692716 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7b5c108130 con 0x7f7b5c0f47c0 2024-01-21T20:08:12.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:12.610+0000 7f7b52ffd700 1 -- 172.21.15.146:0/731692716 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7b5801db70 con 0x7f7b5c0f47c0 2024-01-21T20:08:12.601 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:08:12.601 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:08:12.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:12.612+0000 7f7b64377700 1 -- 172.21.15.146:0/731692716 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f7b4802ff60 msgr2=0x7f7b48032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:12.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:12.612+0000 7f7b64377700 1 --2- 172.21.15.146:0/731692716 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f7b4802ff60 0x7f7b48032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f7b4c0096a0 tx=0x7f7b4c006b40).stop 2024-01-21T20:08:12.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:12.612+0000 7f7b64377700 1 -- 172.21.15.146:0/731692716 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b5c0f47c0 msgr2=0x7f7b5c106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:12.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:12.612+0000 7f7b64377700 1 --2- 172.21.15.146:0/731692716 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b5c0f47c0 0x7f7b5c106700 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f7b58013510 tx=0x7f7b5800b040).stop 2024-01-21T20:08:12.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:12.612+0000 7f7b64377700 1 -- 172.21.15.146:0/731692716 shutdown_connections 2024-01-21T20:08:12.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:12.612+0000 7f7b64377700 1 --2- 172.21.15.146:0/731692716 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f7b4802ff60 0x7f7b48032420 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:12.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:12.612+0000 7f7b64377700 1 --2- 172.21.15.146:0/731692716 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b5c0f47c0 0x7f7b5c106700 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:12.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:12.612+0000 7f7b64377700 1 -- 172.21.15.146:0/731692716 >> 172.21.15.146:0/731692716 conn(0x7f7b5c0f02e0 msgr2=0x7f7b5c0f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:12.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:12.612+0000 7f7b64377700 1 -- 172.21.15.146:0/731692716 shutdown_connections 2024-01-21T20:08:12.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:12.612+0000 7f7b64377700 1 -- 172.21.15.146:0/731692716 wait complete. 2024-01-21T20:08:12.603 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:08:12.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:08:12 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:08:11.212775+0000 mgr.smithi053.fgsmol (mgr.14164) 279 : cluster [DBG] pgmap v265: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:08:13.932 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:08:13 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:08:12.611355+0000 mon.smithi053 (mon.0) 324 : audit [DBG] from='client.? 172.21.15.146:0/731692716' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:08:14.010 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:08:14.010 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:08:14.932 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:08:14 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:08:13.213262+0000 mgr.smithi053.fgsmol (mgr.14164) 280 : cluster [DBG] pgmap v266: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:08:15.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:15.285+0000 7fd995972700 1 -- 172.21.15.146:0/1574824516 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9900f5410 msgr2=0x7fd9900f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:15.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:15.285+0000 7fd995972700 1 --2- 172.21.15.146:0/1574824516 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9900f5410 0x7fd9900f5830 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7fd980009fb0 tx=0x7fd980009b30).stop 2024-01-21T20:08:15.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:15.285+0000 7fd995972700 1 -- 172.21.15.146:0/1574824516 shutdown_connections 2024-01-21T20:08:15.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:15.285+0000 7fd995972700 1 --2- 172.21.15.146:0/1574824516 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9900f5410 0x7fd9900f5830 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:15.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:15.285+0000 7fd995972700 1 -- 172.21.15.146:0/1574824516 >> 172.21.15.146:0/1574824516 conn(0x7fd9900f0b70 msgr2=0x7fd9900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:15.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:15.285+0000 7fd995972700 1 -- 172.21.15.146:0/1574824516 shutdown_connections 2024-01-21T20:08:15.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:15.285+0000 7fd995972700 1 -- 172.21.15.146:0/1574824516 wait complete. 2024-01-21T20:08:15.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:15.286+0000 7fd995972700 1 Processor -- start 2024-01-21T20:08:15.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:15.286+0000 7fd995972700 1 -- start start 2024-01-21T20:08:15.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:15.286+0000 7fd995972700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9900f5410 0x7fd9900fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:15.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:15.286+0000 7fd995972700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9900fb820 con 0x7fd9900f5410 2024-01-21T20:08:15.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:15.287+0000 7fd98effd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9900f5410 0x7fd9900fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:15.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:15.287+0000 7fd98effd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9900f5410 0x7fd9900fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:49774/0 (socket says 172.21.15.146:49774) 2024-01-21T20:08:15.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:15.287+0000 7fd98effd700 1 -- 172.21.15.146:0/3439566709 learned_addr learned my addr 172.21.15.146:0/3439566709 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:08:15.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:15.287+0000 7fd98effd700 1 -- 172.21.15.146:0/3439566709 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd980005040 con 0x7fd9900f5410 2024-01-21T20:08:15.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:15.287+0000 7fd98effd700 1 --2- 172.21.15.146:0/3439566709 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9900f5410 0x7fd9900fd3a0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7fd980013510 tx=0x7fd98000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:15.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:15.287+0000 7fd9777fe700 1 -- 172.21.15.146:0/3439566709 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd980009320 con 0x7fd9900f5410 2024-01-21T20:08:15.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:15.287+0000 7fd995972700 1 -- 172.21.15.146:0/3439566709 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9900fb9c0 con 0x7fd9900f5410 2024-01-21T20:08:15.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:15.287+0000 7fd9777fe700 1 -- 172.21.15.146:0/3439566709 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd980013f00 con 0x7fd9900f5410 2024-01-21T20:08:15.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:15.287+0000 7fd9777fe700 1 -- 172.21.15.146:0/3439566709 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd980014d90 con 0x7fd9900f5410 2024-01-21T20:08:15.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:15.287+0000 7fd995972700 1 -- 172.21.15.146:0/3439566709 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9900fbe50 con 0x7fd9900f5410 2024-01-21T20:08:15.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:15.288+0000 7fd9777fe700 1 -- 172.21.15.146:0/3439566709 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd9800095f0 con 0x7fd9900f5410 2024-01-21T20:08:15.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:15.288+0000 7fd995972700 1 -- 172.21.15.146:0/3439566709 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd990059100 con 0x7fd9900f5410 2024-01-21T20:08:15.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:15.288+0000 7fd9777fe700 1 --2- 172.21.15.146:0/3439566709 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fd97802ff60 0x7fd978032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:15.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:15.288+0000 7fd9777fe700 1 -- 172.21.15.146:0/3439566709 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd980041000 con 0x7fd9900f5410 2024-01-21T20:08:15.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:15.288+0000 7fd98e7fc700 1 --2- 172.21.15.146:0/3439566709 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fd97802ff60 0x7fd978032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:15.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:15.289+0000 7fd98e7fc700 1 --2- 172.21.15.146:0/3439566709 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fd97802ff60 0x7fd978032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fd9840096a0 tx=0x7fd984006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:15.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:15.293+0000 7fd9777fe700 1 -- 172.21.15.146:0/3439566709 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd98001d7f0 con 0x7fd9900f5410 2024-01-21T20:08:15.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:15.496+0000 7fd995972700 1 -- 172.21.15.146:0/3439566709 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd9900fae60 con 0x7fd9900f5410 2024-01-21T20:08:15.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:15.496+0000 7fd9777fe700 1 -- 172.21.15.146:0/3439566709 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd980010230 con 0x7fd9900f5410 2024-01-21T20:08:15.487 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:08:15.487 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:08:15.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:15.498+0000 7fd995972700 1 -- 172.21.15.146:0/3439566709 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fd97802ff60 msgr2=0x7fd978032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:15.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:15.498+0000 7fd995972700 1 --2- 172.21.15.146:0/3439566709 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fd97802ff60 0x7fd978032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fd9840096a0 tx=0x7fd984006b40).stop 2024-01-21T20:08:15.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:15.498+0000 7fd995972700 1 -- 172.21.15.146:0/3439566709 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9900f5410 msgr2=0x7fd9900fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:15.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:15.498+0000 7fd995972700 1 --2- 172.21.15.146:0/3439566709 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9900f5410 0x7fd9900fd3a0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7fd980013510 tx=0x7fd98000b040).stop 2024-01-21T20:08:15.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:15.498+0000 7fd995972700 1 -- 172.21.15.146:0/3439566709 shutdown_connections 2024-01-21T20:08:15.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:15.498+0000 7fd995972700 1 --2- 172.21.15.146:0/3439566709 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7fd97802ff60 0x7fd978032420 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:15.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:15.498+0000 7fd995972700 1 --2- 172.21.15.146:0/3439566709 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9900f5410 0x7fd9900fd3a0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:15.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:15.498+0000 7fd995972700 1 -- 172.21.15.146:0/3439566709 >> 172.21.15.146:0/3439566709 conn(0x7fd9900f0b70 msgr2=0x7fd99005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:15.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:15.498+0000 7fd995972700 1 -- 172.21.15.146:0/3439566709 shutdown_connections 2024-01-21T20:08:15.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:15.498+0000 7fd995972700 1 -- 172.21.15.146:0/3439566709 wait complete. 2024-01-21T20:08:15.490 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:08:15.932 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:08:15 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:08:15.497668+0000 mon.smithi053 (mon.0) 325 : audit [DBG] from='client.? 172.21.15.146:0/3439566709' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:08:16.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:08:16 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:08:15.213674+0000 mgr.smithi053.fgsmol (mgr.14164) 281 : cluster [DBG] pgmap v267: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:08:16.950 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:08:16.950 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:08:18.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:18.129+0000 7f09e86d7700 1 -- 172.21.15.146:0/900073974 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09e00f5410 msgr2=0x7f09e00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:18.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:18.129+0000 7f09e86d7700 1 --2- 172.21.15.146:0/900073974 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09e00f5410 0x7f09e00f5830 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f09dc009fb0 tx=0x7f09dc009b30).stop 2024-01-21T20:08:18.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:18.129+0000 7f09e86d7700 1 -- 172.21.15.146:0/900073974 shutdown_connections 2024-01-21T20:08:18.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:18.129+0000 7f09e86d7700 1 --2- 172.21.15.146:0/900073974 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09e00f5410 0x7f09e00f5830 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:18.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:18.129+0000 7f09e86d7700 1 -- 172.21.15.146:0/900073974 >> 172.21.15.146:0/900073974 conn(0x7f09e00f0b70 msgr2=0x7f09e00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:18.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:18.129+0000 7f09e86d7700 1 -- 172.21.15.146:0/900073974 shutdown_connections 2024-01-21T20:08:18.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:18.129+0000 7f09e86d7700 1 -- 172.21.15.146:0/900073974 wait complete. 2024-01-21T20:08:18.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:18.130+0000 7f09e86d7700 1 Processor -- start 2024-01-21T20:08:18.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:18.130+0000 7f09e86d7700 1 -- start start 2024-01-21T20:08:18.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:18.130+0000 7f09e86d7700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09e00f5410 0x7f09e0105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:18.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:18.130+0000 7f09e86d7700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e0105af0 con 0x7f09e00f5410 2024-01-21T20:08:18.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:18.130+0000 7f09e6473700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09e00f5410 0x7f09e0105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:18.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:18.130+0000 7f09e6473700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09e00f5410 0x7f09e0105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:54814/0 (socket says 172.21.15.146:54814) 2024-01-21T20:08:18.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:18.130+0000 7f09e6473700 1 -- 172.21.15.146:0/3987486280 learned_addr learned my addr 172.21.15.146:0/3987486280 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:08:18.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:18.131+0000 7f09e6473700 1 -- 172.21.15.146:0/3987486280 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09dc005040 con 0x7f09e00f5410 2024-01-21T20:08:18.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:18.131+0000 7f09e6473700 1 --2- 172.21.15.146:0/3987486280 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09e00f5410 0x7f09e0105600 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f09dc013510 tx=0x7f09dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:18.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:18.131+0000 7f09d6ffd700 1 -- 172.21.15.146:0/3987486280 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09dc009320 con 0x7f09e00f5410 2024-01-21T20:08:18.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:18.131+0000 7f09e86d7700 1 -- 172.21.15.146:0/3987486280 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09e0105cf0 con 0x7f09e00f5410 2024-01-21T20:08:18.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:18.131+0000 7f09d6ffd700 1 -- 172.21.15.146:0/3987486280 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f09dc013f00 con 0x7f09e00f5410 2024-01-21T20:08:18.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:18.131+0000 7f09d6ffd700 1 -- 172.21.15.146:0/3987486280 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09dc014d90 con 0x7f09e00f5410 2024-01-21T20:08:18.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:18.131+0000 7f09e86d7700 1 -- 172.21.15.146:0/3987486280 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09e0106180 con 0x7f09e00f5410 2024-01-21T20:08:18.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:18.132+0000 7f09d6ffd700 1 -- 172.21.15.146:0/3987486280 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f09dc0095f0 con 0x7f09e00f5410 2024-01-21T20:08:18.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:18.132+0000 7f09e86d7700 1 -- 172.21.15.146:0/3987486280 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09e0045ad0 con 0x7f09e00f5410 2024-01-21T20:08:18.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:18.132+0000 7f09d6ffd700 1 --2- 172.21.15.146:0/3987486280 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f09cc02ff60 0x7f09cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:18.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:18.132+0000 7f09d6ffd700 1 -- 172.21.15.146:0/3987486280 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f09dc040fe0 con 0x7f09e00f5410 2024-01-21T20:08:18.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:18.132+0000 7f09e5c72700 1 --2- 172.21.15.146:0/3987486280 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f09cc02ff60 0x7f09cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:18.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:18.133+0000 7f09e5c72700 1 --2- 172.21.15.146:0/3987486280 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f09cc02ff60 0x7f09cc032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f09d00096a0 tx=0x7f09d0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:18.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:18.137+0000 7f09d6ffd700 1 -- 172.21.15.146:0/3987486280 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f09dc01d7f0 con 0x7f09e00f5410 2024-01-21T20:08:18.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:18.347+0000 7f09e86d7700 1 -- 172.21.15.146:0/3987486280 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f09e00fae60 con 0x7f09e00f5410 2024-01-21T20:08:18.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:18.347+0000 7f09d6ffd700 1 -- 172.21.15.146:0/3987486280 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f09dc010230 con 0x7f09e00f5410 2024-01-21T20:08:18.338 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:08:18.338 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:08:18.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:18.349+0000 7f09e86d7700 1 -- 172.21.15.146:0/3987486280 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f09cc02ff60 msgr2=0x7f09cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:18.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:18.349+0000 7f09e86d7700 1 --2- 172.21.15.146:0/3987486280 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f09cc02ff60 0x7f09cc032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f09d00096a0 tx=0x7f09d0006b40).stop 2024-01-21T20:08:18.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:18.349+0000 7f09e86d7700 1 -- 172.21.15.146:0/3987486280 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09e00f5410 msgr2=0x7f09e0105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:18.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:18.349+0000 7f09e86d7700 1 --2- 172.21.15.146:0/3987486280 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09e00f5410 0x7f09e0105600 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f09dc013510 tx=0x7f09dc00b040).stop 2024-01-21T20:08:18.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:18.349+0000 7f09e86d7700 1 -- 172.21.15.146:0/3987486280 shutdown_connections 2024-01-21T20:08:18.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:18.349+0000 7f09e86d7700 1 --2- 172.21.15.146:0/3987486280 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f09cc02ff60 0x7f09cc032420 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:18.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:18.349+0000 7f09e86d7700 1 --2- 172.21.15.146:0/3987486280 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09e00f5410 0x7f09e0105600 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:18.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:18.349+0000 7f09e86d7700 1 -- 172.21.15.146:0/3987486280 >> 172.21.15.146:0/3987486280 conn(0x7f09e00f0b70 msgr2=0x7f09e00f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:18.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:18.349+0000 7f09e86d7700 1 -- 172.21.15.146:0/3987486280 shutdown_connections 2024-01-21T20:08:18.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:18.349+0000 7f09e86d7700 1 -- 172.21.15.146:0/3987486280 wait complete. 2024-01-21T20:08:18.341 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:08:18.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:08:18 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:08:17.214024+0000 mgr.smithi053.fgsmol (mgr.14164) 282 : cluster [DBG] pgmap v268: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:08:18.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:08:18 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:08:18.348832+0000 mon.smithi053 (mon.0) 326 : audit [DBG] from='client.? 172.21.15.146:0/3987486280' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:08:19.750 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:08:19.750 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:08:20.932 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:08:20 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:08:19.214421+0000 mgr.smithi053.fgsmol (mgr.14164) 283 : cluster [DBG] pgmap v269: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:08:20.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:20.994+0000 7f30b1c57700 1 -- 172.21.15.146:0/100478263 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30ac0f6510 msgr2=0x7f30ac0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:20.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:20.994+0000 7f30b1c57700 1 --2- 172.21.15.146:0/100478263 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30ac0f6510 0x7f30ac0f6930 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f30a0009fb0 tx=0x7f30a0009b30).stop 2024-01-21T20:08:20.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:20.994+0000 7f30b1c57700 1 -- 172.21.15.146:0/100478263 shutdown_connections 2024-01-21T20:08:20.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:20.994+0000 7f30b1c57700 1 --2- 172.21.15.146:0/100478263 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30ac0f6510 0x7f30ac0f6930 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:20.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:20.994+0000 7f30b1c57700 1 -- 172.21.15.146:0/100478263 >> 172.21.15.146:0/100478263 conn(0x7f30ac0f1c70 msgr2=0x7f30ac0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:20.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:20.995+0000 7f30b1c57700 1 -- 172.21.15.146:0/100478263 shutdown_connections 2024-01-21T20:08:20.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:20.995+0000 7f30b1c57700 1 -- 172.21.15.146:0/100478263 wait complete. 2024-01-21T20:08:20.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:20.995+0000 7f30b1c57700 1 Processor -- start 2024-01-21T20:08:20.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:20.995+0000 7f30b1c57700 1 -- start start 2024-01-21T20:08:20.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:20.995+0000 7f30b1c57700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30ac0f6510 0x7f30ac106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:20.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:20.995+0000 7f30b1c57700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30ac106c20 con 0x7f30ac0f6510 2024-01-21T20:08:20.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:20.996+0000 7f30ab7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30ac0f6510 0x7f30ac106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:20.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:20.996+0000 7f30ab7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30ac0f6510 0x7f30ac106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:54836/0 (socket says 172.21.15.146:54836) 2024-01-21T20:08:20.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:20.996+0000 7f30ab7fe700 1 -- 172.21.15.146:0/863030639 learned_addr learned my addr 172.21.15.146:0/863030639 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:08:20.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:20.996+0000 7f30ab7fe700 1 -- 172.21.15.146:0/863030639 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a0005040 con 0x7f30ac0f6510 2024-01-21T20:08:20.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:20.996+0000 7f30ab7fe700 1 --2- 172.21.15.146:0/863030639 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30ac0f6510 0x7f30ac106730 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f30a0013510 tx=0x7f30a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:20.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:20.996+0000 7f3093fff700 1 -- 172.21.15.146:0/863030639 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30a0009320 con 0x7f30ac0f6510 2024-01-21T20:08:20.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:20.996+0000 7f30b1c57700 1 -- 172.21.15.146:0/863030639 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30ac106e20 con 0x7f30ac0f6510 2024-01-21T20:08:20.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:20.996+0000 7f3093fff700 1 -- 172.21.15.146:0/863030639 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f30a0013f00 con 0x7f30ac0f6510 2024-01-21T20:08:20.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:20.996+0000 7f3093fff700 1 -- 172.21.15.146:0/863030639 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30a0014d90 con 0x7f30ac0f6510 2024-01-21T20:08:20.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:20.996+0000 7f30b1c57700 1 -- 172.21.15.146:0/863030639 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30ac1072b0 con 0x7f30ac0f6510 2024-01-21T20:08:20.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:20.997+0000 7f3093fff700 1 -- 172.21.15.146:0/863030639 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f30a0014420 con 0x7f30ac0f6510 2024-01-21T20:08:20.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:20.997+0000 7f30b1c57700 1 -- 172.21.15.146:0/863030639 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30ac045ad0 con 0x7f30ac0f6510 2024-01-21T20:08:20.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:20.997+0000 7f3093fff700 1 --2- 172.21.15.146:0/863030639 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f309402ff60 0x7f3094032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:20.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:20.997+0000 7f3093fff700 1 -- 172.21.15.146:0/863030639 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f30a00410c0 con 0x7f30ac0f6510 2024-01-21T20:08:20.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:20.997+0000 7f30aaffd700 1 --2- 172.21.15.146:0/863030639 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f309402ff60 0x7f3094032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:20.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:20.998+0000 7f30aaffd700 1 --2- 172.21.15.146:0/863030639 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f309402ff60 0x7f3094032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f309c0096a0 tx=0x7f309c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:20.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:21.002+0000 7f3093fff700 1 -- 172.21.15.146:0/863030639 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f30a00095f0 con 0x7f30ac0f6510 2024-01-21T20:08:21.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:21.204+0000 7f30b1c57700 1 -- 172.21.15.146:0/863030639 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f30ac0fbf60 con 0x7f30ac0f6510 2024-01-21T20:08:21.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:21.204+0000 7f3093fff700 1 -- 172.21.15.146:0/863030639 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f30a001db70 con 0x7f30ac0f6510 2024-01-21T20:08:21.195 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:08:21.195 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:08:21.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:21.206+0000 7f30b1c57700 1 -- 172.21.15.146:0/863030639 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f309402ff60 msgr2=0x7f3094032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:21.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:21.206+0000 7f30b1c57700 1 --2- 172.21.15.146:0/863030639 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f309402ff60 0x7f3094032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f309c0096a0 tx=0x7f309c006b40).stop 2024-01-21T20:08:21.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:21.206+0000 7f30b1c57700 1 -- 172.21.15.146:0/863030639 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30ac0f6510 msgr2=0x7f30ac106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:21.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:21.206+0000 7f30b1c57700 1 --2- 172.21.15.146:0/863030639 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30ac0f6510 0x7f30ac106730 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f30a0013510 tx=0x7f30a000b040).stop 2024-01-21T20:08:21.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:21.206+0000 7f30b1c57700 1 -- 172.21.15.146:0/863030639 shutdown_connections 2024-01-21T20:08:21.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:21.206+0000 7f30b1c57700 1 --2- 172.21.15.146:0/863030639 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f309402ff60 0x7f3094032420 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:21.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:21.206+0000 7f30b1c57700 1 --2- 172.21.15.146:0/863030639 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30ac0f6510 0x7f30ac106730 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:21.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:21.206+0000 7f30b1c57700 1 -- 172.21.15.146:0/863030639 >> 172.21.15.146:0/863030639 conn(0x7f30ac0f1c70 msgr2=0x7f30ac0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:21.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:21.206+0000 7f30b1c57700 1 -- 172.21.15.146:0/863030639 shutdown_connections 2024-01-21T20:08:21.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:21.206+0000 7f30b1c57700 1 -- 172.21.15.146:0/863030639 wait complete. 2024-01-21T20:08:21.198 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:08:21.932 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:08:21 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:08:21.205933+0000 mon.smithi053 (mon.0) 327 : audit [DBG] from='client.? 172.21.15.146:0/863030639' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:08:22.607 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:08:22.607 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 57f6722c-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:08:22.932 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:08:22 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:08:21.214680+0000 mgr.smithi053.fgsmol (mgr.14164) 284 : cluster [DBG] pgmap v270: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:08:23.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:23.830+0000 7f40c61ab700 1 -- 172.21.15.146:0/4229628597 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40c00f51a0 msgr2=0x7f40c00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:23.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:23.830+0000 7f40c61ab700 1 --2- 172.21.15.146:0/4229628597 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40c00f51a0 0x7f40c00f55c0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7f40bc009fb0 tx=0x7f40bc009b30).stop 2024-01-21T20:08:23.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:23.830+0000 7f40c61ab700 1 -- 172.21.15.146:0/4229628597 shutdown_connections 2024-01-21T20:08:23.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:23.830+0000 7f40c61ab700 1 --2- 172.21.15.146:0/4229628597 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40c00f51a0 0x7f40c00f55c0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:23.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:23.830+0000 7f40c61ab700 1 -- 172.21.15.146:0/4229628597 >> 172.21.15.146:0/4229628597 conn(0x7f40c00f09a0 msgr2=0x7f40c00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:23.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:23.830+0000 7f40c61ab700 1 -- 172.21.15.146:0/4229628597 shutdown_connections 2024-01-21T20:08:23.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:23.830+0000 7f40c61ab700 1 -- 172.21.15.146:0/4229628597 wait complete. 2024-01-21T20:08:23.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:23.830+0000 7f40c61ab700 1 Processor -- start 2024-01-21T20:08:23.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:23.831+0000 7f40c61ab700 1 -- start start 2024-01-21T20:08:23.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:23.831+0000 7f40c61ab700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40c00f51a0 0x7f40c00fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:23.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:23.831+0000 7f40c61ab700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c00fb5a0 con 0x7f40c00f51a0 2024-01-21T20:08:23.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:23.831+0000 7f40c51a9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40c00f51a0 0x7f40c00fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:23.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:23.831+0000 7f40c51a9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40c00f51a0 0x7f40c00fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.146:54850/0 (socket says 172.21.15.146:54850) 2024-01-21T20:08:23.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:23.831+0000 7f40c51a9700 1 -- 172.21.15.146:0/3772417475 learned_addr learned my addr 172.21.15.146:0/3772417475 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-21T20:08:23.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:23.831+0000 7f40c51a9700 1 -- 172.21.15.146:0/3772417475 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40bc005040 con 0x7f40c00f51a0 2024-01-21T20:08:23.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:23.831+0000 7f40c51a9700 1 --2- 172.21.15.146:0/3772417475 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40c00f51a0 0x7f40c00fd120 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7f40bc013510 tx=0x7f40bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:23.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:23.832+0000 7f40b5ffb700 1 -- 172.21.15.146:0/3772417475 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40bc009320 con 0x7f40c00f51a0 2024-01-21T20:08:23.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:23.832+0000 7f40c61ab700 1 -- 172.21.15.146:0/3772417475 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40c00fb740 con 0x7f40c00f51a0 2024-01-21T20:08:23.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:23.832+0000 7f40b5ffb700 1 -- 172.21.15.146:0/3772417475 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f40bc013f00 con 0x7f40c00f51a0 2024-01-21T20:08:23.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:23.832+0000 7f40b5ffb700 1 -- 172.21.15.146:0/3772417475 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40bc014dc0 con 0x7f40c00f51a0 2024-01-21T20:08:23.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:23.832+0000 7f40c61ab700 1 -- 172.21.15.146:0/3772417475 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40c00fbbd0 con 0x7f40c00f51a0 2024-01-21T20:08:23.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:23.833+0000 7f40b5ffb700 1 -- 172.21.15.146:0/3772417475 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f40bc0095f0 con 0x7f40c00f51a0 2024-01-21T20:08:23.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:23.833+0000 7f40c61ab700 1 -- 172.21.15.146:0/3772417475 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40c0059100 con 0x7f40c00f51a0 2024-01-21T20:08:23.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:23.833+0000 7f40b5ffb700 1 --2- 172.21.15.146:0/3772417475 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f40ac02ff60 0x7f40ac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:23.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:23.833+0000 7f40b5ffb700 1 -- 172.21.15.146:0/3772417475 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f40bc041100 con 0x7f40c00f51a0 2024-01-21T20:08:23.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:23.833+0000 7f40c49a8700 1 --2- 172.21.15.146:0/3772417475 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f40ac02ff60 0x7f40ac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:23.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:23.834+0000 7f40c49a8700 1 --2- 172.21.15.146:0/3772417475 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f40ac02ff60 0x7f40ac032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f40b00096a0 tx=0x7f40b0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:23.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:23.838+0000 7f40b5ffb700 1 -- 172.21.15.146:0/3772417475 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f40bc014420 con 0x7f40c00f51a0 2024-01-21T20:08:24.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:24.042+0000 7f40c61ab700 1 -- 172.21.15.146:0/3772417475 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f40c00fabe0 con 0x7f40c00f51a0 2024-01-21T20:08:24.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:24.043+0000 7f40b5ffb700 1 -- 172.21.15.146:0/3772417475 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f40bc010230 con 0x7f40c00f51a0 2024-01-21T20:08:24.033 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-21T20:08:24.034 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"57f6722c-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:58:11.102627Z","created":"2024-01-21T19:58:11.102627Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:08:24.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:24.044+0000 7f40c61ab700 1 -- 172.21.15.146:0/3772417475 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f40ac02ff60 msgr2=0x7f40ac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:24.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:24.044+0000 7f40c61ab700 1 --2- 172.21.15.146:0/3772417475 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f40ac02ff60 0x7f40ac032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f40b00096a0 tx=0x7f40b0006b40).stop 2024-01-21T20:08:24.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:24.044+0000 7f40c61ab700 1 -- 172.21.15.146:0/3772417475 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40c00f51a0 msgr2=0x7f40c00fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:24.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:24.044+0000 7f40c61ab700 1 --2- 172.21.15.146:0/3772417475 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40c00f51a0 0x7f40c00fd120 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7f40bc013510 tx=0x7f40bc00b040).stop 2024-01-21T20:08:24.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:24.044+0000 7f40c61ab700 1 -- 172.21.15.146:0/3772417475 shutdown_connections 2024-01-21T20:08:24.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:24.044+0000 7f40c61ab700 1 --2- 172.21.15.146:0/3772417475 >> [v2:172.21.15.53:6800/3203971228,v1:172.21.15.53:6801/3203971228] conn(0x7f40ac02ff60 0x7f40ac032420 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:24.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:24.044+0000 7f40c61ab700 1 --2- 172.21.15.146:0/3772417475 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40c00f51a0 0x7f40c00fd120 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:24.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:24.044+0000 7f40c61ab700 1 -- 172.21.15.146:0/3772417475 >> 172.21.15.146:0/3772417475 conn(0x7f40c00f09a0 msgr2=0x7f40c005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:24.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:24.044+0000 7f40c61ab700 1 -- 172.21.15.146:0/3772417475 shutdown_connections 2024-01-21T20:08:24.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-21T20:08:24.044+0000 7f40c61ab700 1 -- 172.21.15.146:0/3772417475 wait complete. 2024-01-21T20:08:24.036 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-01-21T20:08:24.450 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds 2024-01-21T20:08:24.451 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-01-21T20:08:24.451 DEBUG:teuthology.orchestra.run.smithi053:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-01-21T20:08:24.477 DEBUG:teuthology.orchestra.run.smithi146:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-01-21T20:08:24.495 INFO:tasks.cephadm:Stopping all daemons... 2024-01-21T20:08:24.495 INFO:tasks.cephadm.mon.smithi053:Stopping mon.smithi053... 2024-01-21T20:08:24.495 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-57f6722c-b897-11ee-95b0-87774f69a715@mon.smithi053 2024-01-21T20:08:24.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:08:24 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: cluster 2024-01-21T20:08:23.214974+0000 mgr.smithi053.fgsmol (mgr.14164) 285 : cluster [DBG] pgmap v271: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:08:24.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:08:24 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: audit 2024-01-21T20:08:24.044085+0000 mon.smithi053 (mon.0) 328 : audit [DBG] from='client.? 172.21.15.146:0/3772417475' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:08:24.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:08:24 smithi053 systemd[1]: Stopping Ceph mon.smithi053 for 57f6722c-b897-11ee-95b0-87774f69a715... 2024-01-21T20:08:24.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:08:24 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T20:08:24.672+0000 7fb6797f3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi053 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-01-21T20:08:24.933 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jan 21 20:08:24 smithi053 ceph-57f6722c-b897-11ee-95b0-87774f69a715-mon.smithi053[38842]: debug 2024-01-21T20:08:24.672+0000 7fb6797f3700 -1 mon.smithi053@0(leader) e1 *** Got Signal Terminated *** 2024-01-21T20:08:25.221 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-57f6722c-b897-11ee-95b0-87774f69a715@mon.smithi053.service' 2024-01-21T20:08:25.282 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-21T20:08:25.282 INFO:tasks.cephadm.mon.smithi053:Stopped mon.smithi053 2024-01-21T20:08:25.283 INFO:tasks.cephadm.mon.smithi146:Stopping mon.smithi146... 2024-01-21T20:08:25.283 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-57f6722c-b897-11ee-95b0-87774f69a715@mon.smithi146 2024-01-21T20:08:25.321 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to stop ceph-57f6722c-b897-11ee-95b0-87774f69a715@mon.smithi146.service: Unit ceph-57f6722c-b897-11ee-95b0-87774f69a715@mon.smithi146.service not loaded. 2024-01-21T20:08:25.322 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-01-21T20:08:25.323 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi146" Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi146 with status 5: 'sudo systemctl stop ceph-57f6722c-b897-11ee-95b0-87774f69a715@mon.smithi146' 2024-01-21T20:08:25.324 INFO:tasks.cephadm:Archiving crash dumps... 2024-01-21T20:08:25.326 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/lib/ceph/57f6722c-b897-11ee-95b0-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-01-21_16:37:18-fs-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7525069/remote/ubuntu@smithi053.front.sepia.ceph.com/crash 2024-01-21T20:08:25.326 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /var/lib/ceph/57f6722c-b897-11ee-95b0-87774f69a715/crash -- . 2024-01-21T20:08:25.374 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/lib/ceph/57f6722c-b897-11ee-95b0-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-01-21_16:37:18-fs-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7525069/remote/ubuntu@smithi146.front.sepia.ceph.com/crash 2024-01-21T20:08:25.375 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /var/lib/ceph/57f6722c-b897-11ee-95b0-87774f69a715/crash -- . 2024-01-21T20:08:25.424 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/57f6722c-b897-11ee-95b0-87774f69a715/crash: Cannot open: No such file or directory 2024-01-21T20:08:25.425 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now 2024-01-21T20:08:25.426 INFO:tasks.cephadm:Checking cluster log for badness... 2024-01-21T20:08:25.426 DEBUG:teuthology.orchestra.run.smithi053:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/57f6722c-b897-11ee-95b0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-01-21T20:08:25.460 INFO:teuthology.orchestra.run.smithi053.stderr:grep: /var/log/ceph/57f6722c-b897-11ee-95b0-87774f69a715/ceph.log: No such file or directory 2024-01-21T20:08:25.462 INFO:tasks.cephadm:Compressing logs... 2024-01-21T20:08:25.462 DEBUG:teuthology.orchestra.run.smithi053:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-21T20:08:25.505 DEBUG:teuthology.orchestra.run.smithi146:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-21T20:08:25.532 INFO:teuthology.orchestra.run.smithi053.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-01-21T20:08:25.532 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-01-21T20:08:25.532 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/57f6722c-b897-11ee-95b0-87774f69a715/ceph-mon.smithi053.log 2024-01-21T20:08:25.533 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/57f6722c-b897-11ee-95b0-87774f69a715/ceph-volume.log 2024-01-21T20:08:25.533 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/57f6722c-b897-11ee-95b0-87774f69a715/ceph-mgr.smithi053.fgsmol.log 2024-01-21T20:08:25.533 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/57f6722c-b897-11ee-95b0-87774f69a715/ceph-mon.smithi053.log: /var/log/ceph/57f6722c-b897-11ee-95b0-87774f69a715/ceph-volume.log: /var/log/ceph/57f6722c-b897-11ee-95b0-87774f69a715/ceph-mgr.smithi053.fgsmol.log: 88.3% -- replaced with /var/log/ceph/57f6722c-b897-11ee-95b0-87774f69a715/ceph-volume.log.gz 2024-01-21T20:08:25.534 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/log/rbd-target-api’gzip -5 : No such file or directory--verbose 2024-01-21T20:08:25.534 INFO:teuthology.orchestra.run.smithi146.stderr:-- /var/log/ceph/cephadm.log 2024-01-21T20:08:25.536 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cephadm.log: 96.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-01-21T20:08:25.537 INFO:teuthology.orchestra.run.smithi053.stderr: 91.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-01-21T20:08:25.538 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-01-21T20:08:25.538 INFO:teuthology.orchestra.run.smithi146.stderr:real 0m0.015s 2024-01-21T20:08:25.538 INFO:teuthology.orchestra.run.smithi146.stderr:user 0m0.012s 2024-01-21T20:08:25.538 INFO:teuthology.orchestra.run.smithi146.stderr:sys 0m0.011s 2024-01-21T20:08:25.614 INFO:teuthology.orchestra.run.smithi053.stderr:gzip: /var/log/ceph/57f6722c-b897-11ee-95b0-87774f69a715/ceph-mgr.smithi053.fgsmol.log: file size changed while zipping 2024-01-21T20:08:25.614 INFO:teuthology.orchestra.run.smithi053.stderr: 92.7% -- replaced with /var/log/ceph/57f6722c-b897-11ee-95b0-87774f69a715/ceph-mgr.smithi053.fgsmol.log.gz 2024-01-21T20:08:25.683 INFO:teuthology.orchestra.run.smithi053.stderr: 90.6% -- replaced with /var/log/ceph/57f6722c-b897-11ee-95b0-87774f69a715/ceph-mon.smithi053.log.gz 2024-01-21T20:08:25.684 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-01-21T20:08:25.684 INFO:teuthology.orchestra.run.smithi053.stderr:real 0m0.164s 2024-01-21T20:08:25.685 INFO:teuthology.orchestra.run.smithi053.stderr:user 0m0.241s 2024-01-21T20:08:25.685 INFO:teuthology.orchestra.run.smithi053.stderr:sys 0m0.019s 2024-01-21T20:08:25.685 INFO:tasks.cephadm:Archiving logs... 2024-01-21T20:08:25.685 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-21_16:37:18-fs-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7525069/remote/ubuntu@smithi053.front.sepia.ceph.com/log 2024-01-21T20:08:25.686 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-21T20:08:25.800 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-21_16:37:18-fs-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7525069/remote/ubuntu@smithi146.front.sepia.ceph.com/log 2024-01-21T20:08:25.801 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-21T20:08:25.832 INFO:tasks.cephadm:Removing cluster... 2024-01-21T20:08:25.833 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 57f6722c-b897-11ee-95b0-87774f69a715 --force 2024-01-21T20:09:09.505 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 57f6722c-b897-11ee-95b0-87774f69a715 --force 2024-01-21T20:09:09.812 INFO:tasks.cephadm:Removing cephadm ... 2024-01-21T20:09:09.812 DEBUG:teuthology.orchestra.run.smithi053:> rm -rf /home/ubuntu/cephtest/cephadm 2024-01-21T20:09:09.831 DEBUG:teuthology.orchestra.run.smithi146:> rm -rf /home/ubuntu/cephtest/cephadm 2024-01-21T20:09:09.848 INFO:tasks.cephadm:Teardown complete 2024-01-21T20:09:09.849 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi146 with status 5: 'sudo systemctl stop ceph-57f6722c-b897-11ee-95b0-87774f69a715@mon.smithi146' 2024-01-21T20:09:10.094 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=0ed15adea1244371993c3b88e013995a Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi146 with status 5: 'sudo systemctl stop ceph-57f6722c-b897-11ee-95b0-87774f69a715@mon.smithi146' 2024-01-21T20:09:10.098 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-01-21T20:09:10.109 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-01-21T20:09:10.119 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-01-21T20:09:10.119 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-21T20:09:10.134 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-21T20:09:10.208 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-21T20:09:10.218 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-21T20:09:10.218 DEBUG:teuthology.orchestra.run.smithi053:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-21T20:09:10.221 DEBUG:teuthology.orchestra.run.smithi146:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-21T20:09:10.236 INFO:teuthology.orchestra.run.smithi053.stderr:bash: ntpq: command not found 2024-01-21T20:09:10.237 INFO:teuthology.orchestra.run.smithi146.stderr:bash: ntpq: command not found 2024-01-21T20:09:10.241 INFO:teuthology.orchestra.run.smithi053.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-21T20:09:10.241 INFO:teuthology.orchestra.run.smithi053.stdout:=============================================================================== 2024-01-21T20:09:10.241 INFO:teuthology.orchestra.run.smithi053.stdout:^* hv01.front.sepia.ceph.com 2 6 377 43 -519us[ -510us] +/- 76ms 2024-01-21T20:09:10.242 INFO:teuthology.orchestra.run.smithi053.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 46 -599us[ -590us] +/- 79ms 2024-01-21T20:09:10.242 INFO:teuthology.orchestra.run.smithi053.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 46 +991us[+1000us] +/- 64ms 2024-01-21T20:09:10.242 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T20:09:10.242 INFO:teuthology.orchestra.run.smithi146.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-21T20:09:10.243 INFO:teuthology.orchestra.run.smithi146.stdout:=============================================================================== 2024-01-21T20:09:10.243 INFO:teuthology.orchestra.run.smithi146.stdout:^* hv01.front.sepia.ceph.com 2 6 377 44 -521us[ -515us] +/- 76ms 2024-01-21T20:09:10.243 INFO:teuthology.orchestra.run.smithi146.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 44 -569us[ -563us] +/- 79ms 2024-01-21T20:09:10.243 INFO:teuthology.orchestra.run.smithi146.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 44 +978us[ +984us] +/- 64ms 2024-01-21T20:09:10.243 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T20:09:10.244 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-21T20:09:10.253 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-21T20:09:10.253 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-21T20:09:10.268 DEBUG:teuthology.orchestra.run.smithi053:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-21T20:09:10.348 DEBUG:teuthology.orchestra.run.smithi146:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-21T20:09:10.409 DEBUG:teuthology.orchestra.run.smithi053:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-21T20:09:10.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T20:09:10.440 DEBUG:teuthology.orchestra.run.smithi146:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-21T20:09:10.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T20:09:10.469 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-21T20:09:10.478 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-21T20:09:10.487 INFO:teuthology.task.internal:Duration was 1237.218382 seconds 2024-01-21T20:09:10.488 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-21T20:09:10.499 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-21T20:09:10.500 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-21T20:09:10.502 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-21T20:09:10.547 INFO:teuthology.orchestra.run.smithi053.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-21T20:09:10.554 INFO:teuthology.orchestra.run.smithi146.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-21T20:09:10.946 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-01-21T20:09:10.946 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi053.front.sepia.ceph.com 2024-01-21T20:09:10.946 DEBUG:teuthology.orchestra.run.smithi053:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-21T20:09:10.970 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi146.front.sepia.ceph.com 2024-01-21T20:09:10.970 DEBUG:teuthology.orchestra.run.smithi146:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-21T20:09:10.994 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-01-21T20:09:10.994 DEBUG:teuthology.orchestra.run.smithi053:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-21T20:09:11.012 DEBUG:teuthology.orchestra.run.smithi146:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-21T20:09:11.084 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-01-21T20:09:11.085 DEBUG:teuthology.orchestra.run.smithi053:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-21T20:09:11.087 DEBUG:teuthology.orchestra.run.smithi146:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-21T20:09:11.175 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-21T20:09:11.184 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-21T20:09:11.185 DEBUG:teuthology.orchestra.run.smithi053:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-21T20:09:11.212 DEBUG:teuthology.orchestra.run.smithi146:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-21T20:09:11.239 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-21T20:09:11.251 DEBUG:teuthology.orchestra.run.smithi053:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-21T20:09:11.254 DEBUG:teuthology.orchestra.run.smithi146:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-21T20:09:11.279 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern = core 2024-01-21T20:09:11.305 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern = core 2024-01-21T20:09:11.338 DEBUG:teuthology.orchestra.run.smithi053:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-21T20:09:11.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T20:09:11.369 DEBUG:teuthology.orchestra.run.smithi146:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-21T20:09:11.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T20:09:11.395 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-21T20:09:11.408 INFO:teuthology.task.internal:Transferring archived files... 2024-01-21T20:09:11.409 DEBUG:teuthology.misc:Transferring archived files from smithi053:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-21_16:37:18-fs-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7525069/remote/smithi053 2024-01-21T20:09:11.410 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-21T20:09:11.451 DEBUG:teuthology.misc:Transferring archived files from smithi146:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-21_16:37:18-fs-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7525069/remote/smithi146 2024-01-21T20:09:11.452 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-21T20:09:11.490 INFO:teuthology.task.internal:Removing archive directory... 2024-01-21T20:09:11.491 DEBUG:teuthology.orchestra.run.smithi053:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-21T20:09:11.493 DEBUG:teuthology.orchestra.run.smithi146:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-21T20:09:11.542 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-21T20:09:11.553 INFO:teuthology.task.internal:Not uploading archives. 2024-01-21T20:09:11.553 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-21T20:09:11.568 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-21T20:09:11.568 DEBUG:teuthology.orchestra.run.smithi053:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-21T20:09:11.570 DEBUG:teuthology.orchestra.run.smithi146:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-21T20:09:11.586 INFO:teuthology.orchestra.run.smithi053.stdout: 393225 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 21 20:09 /home/ubuntu/cephtest 2024-01-21T20:09:11.599 INFO:teuthology.orchestra.run.smithi146.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 21 20:09 /home/ubuntu/cephtest 2024-01-21T20:09:11.600 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-21T20:09:11.610 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-21T20:09:11.641 INFO:teuthology.nuke:Checking targets against current locks 2024-01-21T20:09:11.664 DEBUG:teuthology.nuke:shortname: smithi053 2024-01-21T20:09:11.664 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-21T20:09:11.688 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi053.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-21_16:37:18-fs-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7525069', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-21 19:39:41.954777', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKpOsOjovXAabvq3jBF/C3fX8dBGkOjGUi3rKmXmWHvlzOiZgPGWaB05cvinr0wkjK+mL2LXbbaLTbta922SPeA='} 2024-01-21T20:09:11.689 DEBUG:teuthology.nuke:shortname: smithi146 2024-01-21T20:09:11.689 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-21T20:09:11.712 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi146.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-21_16:37:18-fs-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7525069', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-21 19:39:41.956506', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD3QXMV9nD31+bSTFbZE+8Kvev917q1qJtUgAKnX7s/UtLVRVFfrd0QefK8rG/D6DvyvClaX7OrgeJnt77fF0UQ='} 2024-01-21T20:09:11.741 INFO:teuthology.orchestra.console.smithi053:Power off 2024-01-21T20:09:11.741 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-21T20:09:11.757 INFO:teuthology.orchestra.console.smithi146:Power off 2024-01-21T20:09:11.758 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-21T20:09:11.767 DEBUG:teuthology.orchestra.console.smithi053:power off output: Chassis Power Control: Down/Off 2024-01-21T20:09:11.768 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T20:09:11.778 DEBUG:teuthology.orchestra.console.smithi146:power off output: Chassis Power Control: Down/Off 2024-01-21T20:09:11.779 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T20:09:11.788 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is on 2024-01-21T20:09:11.800 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is on 2024-01-21T20:09:15.789 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T20:09:15.801 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T20:09:15.900 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is on 2024-01-21T20:09:15.909 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is on 2024-01-21T20:09:19.907 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T20:09:19.915 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T20:09:20.018 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is off 2024-01-21T20:09:20.025 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is off 2024-01-21T20:09:20.119 INFO:teuthology.orchestra.console.smithi053:Power off completed 2024-01-21T20:09:20.126 INFO:teuthology.orchestra.console.smithi146:Power off completed 2024-01-21T20:09:20.230 INFO:teuthology.run:Summary data: description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 1237.2183816432953 failure_reason: 'Command failed on smithi146 with status 5: ''sudo systemctl stop ceph-57f6722c-b897-11ee-95b0-87774f69a715@mon.smithi146''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=0ed15adea1244371993c3b88e013995a status: fail success: false 2024-01-21T20:09:20.231 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-21T20:09:20.304 INFO:teuthology.run:FAIL